vogon poetry

Google Chrome Keyword Shortcuts

2011-06-01T18:48:00.003-05:00

I do like Google Chrome and that it has merged the functionality of a browsers address box with the default search engine box, calling it the omnibox. Occasionally it parses my search query into a URL, so I have added what Chrome calls a Search Engine, but what is more commonly called a keyword shortcut. You can add any keyword shortcut that you want to the omnibox. To ensure a query in the omnibox is parsed as a query and not a URL I created the gs (google search) shortcut. On the Google Chrome preferences page select the Manage Search Engine button. You should see something like the image below.

To add the gs keyword click on the Add a new search engine box, and give the search a name, like web, click on the Keyword box and enter a your keyword for a Google search, I used gs. Finally click on the URL with %s in place of query box and enter the URL https://encrypted.google.com/search?q=%s then hit return. As you hover the mouse over these entries a light blue color Make Default box will appear, click it to set this as your default search engine if you want.

I have about ten keyword shortcuts I use in Google Chrome, but the one I use most often is the I'm Feeling Lucky shortcut. I call it the goto shortcut; the URL for the shortcut is https://encrypted.google.com/search?q=%s&btnI=Im+Feeling+Lucky and I use the semicolon character ; for its keyword since it is quick to type.
The list below gives the use and URL for some of my other shortcuts:

definitions - https://encrypted.google.com/search?tbs=dfn%3A1&q=%s
recipes - https://encrypted.google.com/search?tbs=rcp%3A1&q=%s
gmail - https://mail.google.com/mail/#apps/%s
images - https://encrypted.google.com/search?tbm=isch&q=%s

Unfortunately the Google Chrome Sync feature does not sync the Search Engines, so you will need to add to each computer you use Chrome on.

Suing Craigslist

2009-03-05T18:33:00.003-05:00

The latest news about Sheriff Thomas Dart suing Craigslist because he claims it is the "largest source of prostitution" seems quite absurd. Since he is a Sheriff I can't help but think he is too entrenched in the problem to realize the possibility that Craigslist isn't generating more prostitution, but merely exposing how much really exists. Also, the quote "Dart claimed the traffic it generates accounts for the bulk of the web site's popularity" is quite an outrageous claim without an analysis of the web logs. Seems like a classic case of "shoot the messenger". Next, will he sue car manufactures because so many automobiles are used to commit crimes?

All of this said, my bet is that Sheriff Dart is just being politically savvy. By suing Craigslist and making such unsubstantiated claims he is drawing attention to the problem that will hopefully build public pressure to see that the Sheriff's department gets more taxpayer funding to attack the problem.

Serendipity of Search

2008-10-04T20:48:00.004-05:00

Erin McKean's TED talk, Redefining the dictionary, is excellent. Her comment:

... in fact online dictionaries replicate almost all the problems of print, except for searchability. And when you improve searchability you actually take away the one advantage of print, which is serendipity. Serendipity is when you find things you weren't looking for because finding what you are looking for is so damn difficult.

reminded me of popular quote by Franklin P. Adams:

I find that a great part of the information I have was acquired by looking up something and finding something else on the way.

GOOG-411

2008-09-16T21:48:00.003-05:00

I really like the GOOG-411 service and have come to use it instead of looking up numbers online. However, today my daughters unintentionally launch a local denial of service attack against me as I used it. I called to place an order at our local zpizza (yes, just like their demo video ). As it was giving me my one selection my oldest was talking to her sister about wanting "to go back to ballet", upon which the service went back to the top menu. I didn't pick up on this at first and was completely confused as to why the system was jumping around without me saying anything, but eventually stepped out of the car into the quite outdoors to complete my order. I remember reading InfoWorlds interview with Marissa Mayer about how the goal of the service is to build a training set for speech models. She states:

You may have heard about our [directory assistance] 1-800-GOOG-411 service. Whether or not free-411 is a profitable business unto itself is yet to be seen. I myself am somewhat skeptical. The reason we really did it is because we need to build a great speech-to-text model ... that we can use for all kinds of different things, including video search.
The speech recognition experts that we have say: If you want us to build a really robust speech model, we need a lot of phonemes, which is a syllable as spoken by a particular voice with a particular intonation. So we need a lot of people talking, saying things so that we can ultimately train off of that. ... So 1-800-GOOG-411 is about that: Getting a bunch of different speech samples so that when you call up or we're trying to get the voice out of video, we can do it with high accuracy.

This is such a brilliant idea. I really can't think of a better way to get a huge group of people to help associate the words they pronounce with its text equivalent at such a low cost (to Google). But I also wondered about the bias they would need to deal with in the training set. When I use the service I do find myself pronouncing the words more clearly, also the distribution of phonemes in place and business names might be significantly different from common words. I suspect that these are minor issues that are mitigated as the data set grows larger.

One Step Closer ...

2008-09-14T20:37:00.003-05:00

Yesterday I showed my support for the Howard County Autism Society by participating in their One Step Closer walk around Centennial Lake, in Columbia, Maryland. My daughters wanted to also partake but couldn't make it, so I took some photos of the educational signs along the footpath with my iPhone during the walk to show them. I emailed the photos to my Flickr page as I took them. Unfortunately, when I emailed them the geo-location information was not included or ignored by Flickr. However, when I upload them via my computer the geo-location information was included and automatically mapped by Flickr. This is the first time I experienced this technology, and I'm amazed at the accuarcy of the geo-coordinates, the photos on the Flickr map follow the footpath very well. Every camera should have a GPS!

Google News Queries Returning Foreign Language Results

2008-08-29T20:37:00.005-05:00

Mahesh KM's post to the Google News blog states:

Occasionally, sources in other languages may have extremely relevant results for your query. With this feature, we want to offer stories from these sources to you when they're helpful.

So if I type a query in English I may get very relevant results in Chinese? Mahesh points out that since there are so many English sources of news it is rather unlikely to get a foreign language result with an English query. One of the examples is searching for news about the G8 Summit from the Spanish news version. I'm surprised they did not include a link to Google-translate the foreign language results into the language of the query. Perhaps I am reading too much into this feature. I interpret it to mean that they are translating each news article indexed into various languages and returning the most relevant from the original and translated articles. Anyone know if this is the case?

ETech07: Flickr Photo Links

2007-04-04T20:17:00.000-05:00

ETech 2007 Flickr photos sorted by interestingness or most recent.

ETech07: Applying Game Mechanics to Social Software

2007-04-04T20:04:00.000-05:00

Amy Jo Kim of ShuffleBrain gave an excellent tutorial on ... Applying Game Mechanics to Social Software. She is also the author of the book Community Building on the Web. First, she emphasized thinking of the user as a player and participant, not just a user. She started with a few definitions of the word game and settled on "a structured experience with rules and goals that's fun". For example, ebay and flickr can be viewed as a game in this definition. Why are games good at driving behavior? Because well designed games are fun to play and it is a basic human need to have fun and play. The best designed games enable a player to engage in flow - "...an experience that is at once demanding and rewarding--an experience that ... is one of the most enjoyable and valuable experiences a person can have" (from an Amazon review). What are game mechanics? Features that engage a player, or make a game fun and compelling. The five basic components of games mechanics are collecting, points, feedback, exchanges and customization. We all collect things (I can't help but think of George Carlin's A Place for My Stuff, Skit). Giving the player the ability to collect things and enabling them to show their collection gives the player "bragging rights"; it can also encourage them to complete a set if the possible contents are predefined, e.g., baseball cards, or habbohotel cards. Points are common to most games but are better when you can "redeem" them for something, e.g., a discount coupon, frequent flyer miles, better support (ebay), etc... Once you have a point-system you can create leaderboards, like at youTube. But choose your leaderboard with care, what is displayed will express your sites values (which might not be what you hoped), and it will be gamed by some players. One way around this is to have multiple leaderboards (most viewed, most comments, highest rated, etc...) or even statistical ranking leaderboards (flick explore) that are harder to game. Also, points enable you to have levels, which are really just categories for accumulated points, and are better when rewarded for being attained, for example, ebay has a star/color levels. In role playing games they usually unlock new powers. They do the same on many social sites, for example, on ebay one can become a power seller once you get enough good ratings. Even credit card companies do this, the more you buy and the better customer you are, the faster your phone calls are answered. Feedback helps players learn to master the skills needed to play on the site and it should make the site more fun -- it should make mundane tasks fun. The ability for exchanges (structured social interactions) between players should be available whether the exchanges come about due to explicit rules (like chess) or implicitly (commenting on each others photos on flickr). Players should have the ability to customize, it increases investment by the player. A mySpace user home page is a great example of customization as are second life characters. Related article: The Core of Fun: Ralph Koster. Related blog posts: google, technorati.

ETech07: Kathy Sierra's Canceled Tutorial -- Bummer

2007-03-27T17:51:00.000-05:00

I am at ETech and was really looking forward to Kathy Sierra's tutorial on Creating Addictive User Experiences but she had to cancel due to stalking and death threats. I truly hope the guilty parties are subdued so she feels safe to come back to blogging and even the outside world.

ETech07: Coder to Co-Founder: Entrepreneuring for Geeks

2007-03-27T13:59:00.000-05:00

I attended Marc Hedlund's Coder to Co-Founder: Entrepreneuring for Geeks at ETech today. An excellent tutorial about what it takes to start a web company. He covered a lot of ground that I can only highlight. First, what is an entrepreneur? Someone who creates something from nothing. What is that something? The something starts with the idea that won't leave you alone. Even if the idea keeps getting the "that will never fly" response when you tell others don't given up, evolve it into something better (reminded me of how Fred Smith got a low grade on his business-idea for FedEx in college). However, know when to move onto a new one. Your ideas (engineering) will get better the more you know about business, marketing, sales, etc... If you have no education in business get some. You don't need a MBA, just read some books on it or take a short course on business administration. When do you know if you are ready to try? When you are more excited about your idea than afraid to pursue it -- if you not afraid at all you probably shouldn't start. What is it like to be an entrepreneur? It sounds like a wild ride. It's fun to be your own boss but it can also be very hard; everyday is something new, usually involving things you have never done before. But don't become one to get rich or become famous; you need your heart in the idea, otherwise you will probably give up early. Remember, losing sucks, especially if you lose other peoples money who also believed in your idea. How do you get started? First, build within what you know and build a prototype (or even a picture) of the interface and give it to people for feedback, it is a great momentum builder. But remember, give people what they need, not what they say they want. Share you ideas, you need feedback to improve it. Get a partner (no more then two) so you can feed off each other; it helps a lot to keep the momentum building. Don't worry about timing, the market fluctuates and you need to just adjust with it. (Now it is hard to afford office space and find good programmers, but relatively easy to get funding.) Don't let competition scare you off, emphasize what you can do better than them. Do worry about whether there will be demand for you product/service. Go out and find people who would be your customers (get non-computer people). How long does it take to explain your idea? Do they become excite? Learn how to explain it to them using common terms. Have a friend deliver your pitch to someone and see how it comes off. Also, imagine how you would compete with yourself, i.e., how would you compete with what you just launched. What about co-founders? It will change your relationship with the person if you know them, regardless of whether the idea fails or succeeds, so keep that in mind. Having a co-founder is good because you can build momentum off each other; but having more four or more is nuts, the decisions get harder with more. Regardless of the number of co-founders work out a good way to handle disagreements. Also the first three (or two) hires will set the tone for the company, so choose them very wisely. In fact, hire them as contractors first to see if they work out. However try to get contractors who are full time so their attention is undivided. Also, don't ignore good people who can't commit to your company until you have better backing, they just can't afford the risk since they have a family, etc... It is really important to hire people you like, who like you, and who are nearly as passionate about your idea as you are. If you have never started a company before, you will need bootstrapping funding, i.e., your own or family or friends money. Best way is moonlighting, work on it in the evenings and on weekends. Tough to do but you keep full ownership of the company; you will probably worry a lot about money. Once you having something running you can seek investment from venture capital. The best way to try to get investment is to not need the money to survive, but to grow or to grow faster. Funding is great because you can start paying your employees! The more you get the more strings are attached. Angel funding is from 0.5-1.5 million with you giving up 5-15% of the company. Series A is 1-10 million with you giving up 40-50% of the company and getting a board member. When giving a sales pitch to a VC you need 10-15 slides, a two or three page execute summary, and an introduction. Keep the pitch plainspoken and entertaining. Remember, they invest in companies, not products. Also, VC leave the planet in August and December so don't even try to reach one. Marc then went into two case studies, gripejuice and wesabe. Gripejuice was to be a social site to log problem transactions customers had with companies. He didn't peruse it for many reasons, one being the effort on the user to was too great. However, a similar site has launched called 321-CALL-LOG. The other is a social finance site called wesabe that recently launched. The biggest obstacle to overcome for the business model was addressing the privacy concerns the user would have with uploading or financial records with the site. He overcame thus by enabling user to anonymize the data. His recommended reading list: The Art of the Start, by Kawasaki, Founders at Work, by Livingston, Getting Real by 37signals, The Ten Day MBA by Silbiger, Product Marketing Handbook for Software by Chapman, and What the Numbers Say, by Boyum.

OSCON06: Taking a Closed Source Product Open

2006-08-12T11:13:00.001-05:00

Neelan Choksi gave a talk at OSCON entitled "Lessons Learned in Taking a Closed Source Product Open" about BEA making Kodo (Java persistence APIs) an open source project. Choksi is the senior director of products at BEA. Why did he work at making Kodo open source? The users kept asking them to do it and BEA can make more by open sourcing it by selling support and add-ons. He had some good advice about the whole process. First, note that it is a one way process, once you go open source there is not turning back. Company leadership buy-in is a must, because you need your whole organizations support. Even though the project is open sourced it will still need management, marketing, documentation, and quality assurance testing. Good documentation is very important for an open source project, it helps make it easier for developers to decide if they want to join and where they can contribute. One goal is to reduce market costs by building a community around the product. To do this you need a really good evangelizer, someone who can hit the speaker circuit. Ironically it probably costs more initial to open source a close project than starting one from scratch. A lot of effort has to be devoted to getting the documentation up to a level that will help build a community. Make sure your support organization will be able to handle the increased load, especially if you offer some low cost support to get people to seriously try out the software. Lastly, don't expect results overnight, make sure your management understands that it will take at least a year or more to build up a community.

OSCON06: So, You Want to Build an Open Source Community...

2006-08-11T18:51:00.000-05:00

J Aaron Farr gave a talked at OSCON titled "So, You Want to Build an Open Source Community: Learning from Apache" about, obviously, building an open source community and a bit about Apache Incubator. First, community is the key to any open source project (this was restated in about every talk I attended at OSCON). It's not just about having a community of developers and users but about growing and evolving them so that all the developers are also users and many of the users become involved more directly. It reminded me of how Firefox has a lot more than just software developers, they have graphic artists, writers, lawyers, marketers, etc... contributing to the project. He emphasize that commiters (contributers) should eat their one dogfood. In other words, the developers and users should really believe in the project. Growth of the community is really needed to ensure the stability of the project. If you have a stable community then as people come and go others can continue it. So what do you need to build the community? At a minimum working code that developers can contribute to and a good means for the community to communcate. Like a good web site with documentation, mailing lists, newsgroups, an issue tracking system, and a source code repository. Once you have all this you really need a clear pathway for people to see where they can contribute; like a web site that makes easy for people to see how they could contribute to the project and how to become a member. He also discussed the Apache Incubator, which is a process the Apache Foundation setup to review projects for inclusion into the Foundation. The slides for his talk are here and his blog is CubicleMuses.

OSCON06: Open Technology Development

2006-08-10T20:59:00.000-05:00

John Scott's talk on Open Technology Development: Open Source and the U.S. Government was about how to change the culture in DoD and the U.S. Government in general to foster better software development, be it in-house or contracted development. Not surprisingly, canvassing DoD he found huge interoperability problems, lots of re-invention, lots of failed projects, and that those in the field use whatever it takes to get the job done rather than try and do it by the book. This is entirely a problem of policy, not a lack of technology. So DoD is looking to the open source software model to provide a faster development process. The current acquisition process for DoD software fosters interoperability, stifles innovation, and is far from cost effective (for all the contracts DoD has with PeopleSoft they could have acquired the company). Worse, many opponents "out develop" the DoD because the process is so slow. So how does one go about creating a community that develops software collaboratively across all of DoD? First, you rename "open source" to "open technology development" so as not to scare those afraid of open source. Next, you need a policy that allows and even encourages it. (The part he is currently working on.) It can be done, there are some isolated projects that DoD has contributed a lot too such as openssl, geo-spatial tools, solid modeling, and secure linux. Scott co-authored a report entitled OTD Road Map that outlines why and how DoD should create a DoD open source software development community. On 14 September 2006 there is a conference by the Association for Enterprise Integration on Open Source New DoD Paradigm, or Business as Usual?. Some other related sites are Open Technology Development and Governement Forge.

OSCON06: Asymmetric Competition

2006-08-03T20:49:00.000-05:00

Tim O'Reilly interviewed Jim Buckmaster (ceo, craigslist) about craigslist. Jim pointed out the craigslist is starting to charge a samll fee for some real estate listings. However, this was at the users request to cut down on spam and repeated listings. Tim asked him if he felt guilty sucking so much revenue out of newspapers? His answer: "No, they still make a profit." Tim asked about future plans. Jim stated they don't need to do something new, they just need to do a lot more of what the users ask for. An interesting blog about craigslist is on the O'Reilly radar here. Photo courtesy of James Duncan Davidson/O'Reilly Media.

OSCON06: What's Microsoft Doing with Open Source?

2006-08-03T17:57:00.000-05:00

Danese Cooper interviewed Bill Hilf (general manager of platform strategy, microsoft ) on "What's Microsoft Doing with Open Source." Bill started by pointing out that microsoft does support the open source community via conferences like OSCON and Port25. Over the past three to four years microsoft has gone from fearing open source (linux) to becoming a lot smarter about it. To learn from open source microsoft has to engage it and get involved. Bill's team is composed of about 20-30 in Redmond and another 100 people worldwide, compared to about 60,000 total at microsoft. The goal is to try to influence by spending about 75% of their time with the production teams. Yet microsoft still really needs a governance model. Cooper: Now that Martin Taylor left is that helping you in your job? Hilf: He did help centralize the company on the same page. Cooper: But he did fund a lot of disinformation about open source? Hilf: We never had a disinformation campaign. The message format needed to be better (the audience laughed a lot at that answer). Danese and Bill discussed the plug-ins being developed to store microsoft office documents in an openXML format. Cooper commented that microsoft is still not providing enough information to the project for the plug-ins to accurately convert documents between openXML and the ODE format. Bill really was not up-to-date on how well the plug-ins are working. Cooper: What is the hardest part of your job? Hilf: Dealing with the closed mindedness on both sides, (at microsoft and in the open source community). Cooper: Do you thing the European Union did a better job of teaching microsoft to play fair then DOJ? Hilf: The DOJ ruling had quite an effect on microsoft. Some questions were taken from the audience near the end. Kartik Subbarao asked how can microsoft be trusted (in the open source community) and Bill gave the best answer possible "By consistent action over time going in the right direction." Overall Bill Hilf did a "great" job of answering the questions, but I was kind of surprised he wasn't more knowledgeable about the state of the openXML project. Photo courtesy of James Duncan Davidson/O'Reilly Media.

OSCON06: Ghost in the Machine

2006-08-02T18:28:00.000-05:00

At the "Ghost in the Machine" panel discussion Tim O'Reilly essentially asked Jeremy Zawodny (developer relations, yahoo), Chris DiBona (open source program manager, google), and Jim Buckmaster (CEO, craigslist) how do they use open source and how do they contribute to open source. First, all three basically said they would not exist without open source. Jeremy said yahoo uses freeBSD, linux, C, C++, PHP, Perl, and Python on its servers and the desktops are "across the board". Chris said google uses linux, mysql, gcc, g++, and python, but not a lot of apache. They have developed their own web server for better throughput. Jim said that craigslist uses the LAMP model (linux, apache, mysql, perl) and ubuntu on the desktops. How do you contribute to the open source community? First, they all have developers who submit patches to many different open source projects. Second, yahoo and google have hired many major contributors to open source projects who have the job of continuing the development of the project; but they also make enhancements that are not rolled out because it gives them a competitive advantage. Also, google and yahoo have free APIs and all three make monetary contributions to open source projects. Why don't you contribute more of your code to the open source community? All three said something akin to 1) the code is not up to the standard of open source code (it would be embarrassing), 2) the code is what sets us apart from our competitors, 3) the code would be useless without the underlying infrastructure, 4) moving from closed source to open source can take a lot of resources so its hard to determine what would be best to open source, and 5) we don't want spammers learning how we find spam. Also, Jim Buckmaster has to be the most laid-back CEO of a successful company I have ever seen. Photo courtesy of James Duncan Davidson/O'Reilly Media.

OSCON06: Face 2 Face: Processes for OS Communities

2006-08-01T20:26:00.000-05:00

Ever attended a meeting that you felt just wasn't focusing on the real issues and knew others at the meeting felt the same. Kaliya Hamlin's "Face 2 Face: Processes for OS Communities" tutorial at OSCON 2006 provided good techniques to make meetings and conferences a lot more productive. While her tutorial was targeted at a conference to addresses problems within an open source community, they apply to just about any size gathering of people invited to resolve some issues. Some of the methods are obvious while others would be considered avant-garde. Also, at the request of many participants, we discussed ways to deal with people who derail meetings, or worse, get verbally confrontational or blow-up. First, an important part of any meeting, that is often overlook, is to let everyone introduce themselves and if time permits provide more than just their name. Obviously this is for groups having people who don't know each other by name (yet). It is really important for people to put a face to name. At the level of a conference or large meeting she emphasized that the "art of invitation" is really important in getting the right people to attend. Express your hopes and goals of the meeting in the invitation. Also, if it is for a large conference try to come up with a cool name or logo, some examples are winecamp and the internet identity conference. Next, at the start of the conference get participants to suggest sessions on a sheet of paper and create a chart of the available rooms and time slots. Let everyone post his or her sessions to the chart. Once all have seen the suggestions ask people to negotiate a better arrange and/or come up with better sessions. We did this pretending to attend a conference on ways to "improve online collaboration". I suggested a session entitled "ways to reduce office email using a community blog". About three others had similar session ideas and we quickly rearranged and merged our sessions into a topic track we all like. I was somewhat surprising how smoothly this went, I expected some chaos to ensue but none did. To "break the ice" of a meeting we played the "game" called "A Strong Wind Blows..." akin to musical chairs. Everyone sits in a circle except for one person standing in the center. The center person states a quality about themselves like "a strong wind blows for all linux users". Those who use linux get up and scramble for a new open seat, including the center person. The one left standing then takes a turn. I have to admit I had some reservations at first but was amazed at how well it worked. There was quite a bit more openness and energy in the room after only about 10 rounds. I wouldn't try this technique at the next board meeting; however, it really seemed perfect for an offsite. It is a technique to get people to know one another and get a sense of who is in the room. After this she outline the "fishbowl technique". In this method a set of chairs are arranged in concentric circles starting with about seven in the center. Get six volunteers to sit in the center or choose six with differing opinions about an issue being addresses. During the conversation among the six when someone outside has something to add they sit in the empty chair and one of the others self-selects themselves to step out. The method worked well for our discussion, how to deal with someone who blows-up at or derails a meeting. Some of the suggestions were "work hard not to hire such people", "have some formal communication methods", "everyone should take a break and the manager should talk to the person privately". When talking with such a person emphasize connection, build up their trust, and that you are giving them the attention they are asking for and they now need to give others equal time. When listening don't always do an analysis (male trait). (read Six Thinking Hats by Edward de Bono). Try to make the environment as open as you can to allow everyone to comment. If you find a meeting spiraling into defensive communications, e.g., hearing a lot of "I don't think …" or "no that's not …" try to pull it out possibly by pointing out the defensive patterns observed. If you find someone keeps shooting down ideas create a forum (meeting, community-blog, etc…) of some kind having the sole purpose of generating ideas and forbid any negative comments for a period of time (a week or two) after an idea is posted. Another issue raised was how to handle someone who doesn't follow through on tasks? Create a list of action items at the meeting and check-up on each person about their task. If they have not completed it ask if they still "want to own it". If not, move it to the orphaned list. At the next meeting review the list, if important items are on it they will be addressed. The other technique we did was called spectrogram and is meant to get the range of opinions on a critical issue. We choose the question "Can you trust Microsoft?" A line of tape on the floor represented the range from "yes" to "no" with the middle being "don't know". You stand on the line where your opinion lies. (I was alone at the "no" end, surprising since this was at OSCON.) Most people hovered around the middle. Next, we had someone ask people why they chose to stand at this point on the line. It was enlightening to here other points of view. A lot of people seemed the feel that Microsoft is changing for the better these days. Kaliya had another technique called speed-geeking that she wanted us to try but time ran out. Overall it was good excellent session and I learned a lot. Benjamin Smedberg also blogged about the session.

OSCON06: 7 Principles of Better API Design

2006-07-24T23:55:00.000-05:00

Damian Conway gave a good tutorial on 7 Principles of Better API Design at OSCON 2006. One of the things that struck me during the tutorial is how much the principles parallel human interface design; which really shouldn't have been too surprising since it's people who use an API. The overall message was keep it simple -- easy to use and easy to understand. The first principle was 'Do One Thing Really Well'. His example of this was the Perl6::Say module with the 'say' function that automatically appends a new line to the item printed. I had gotten into the habit of declaring 'our($el)="\n"' in a module and using 'print .....,$el;'. Obviously 'say' is a much better way of handling this. Another example was Perl6::Slurp to read in the entire contents of a file. Some bads ways to do this are '$text=join("", <$fh>);' and worse this '$text.=$_ for (<$fh>)'. This can all be done better with '$text=slurp $f" where $f is either a file handle or the path to a file. A much more elegant solution. The second principle is 'Design by Coding' -- he is not implying extreme programming here. A more descriptive title might be 'Design by Sample Re-coding'. Write some code that would use the modules and write the interfaces as you would like them. Now, iterate this process with the third principle, "Evolve by Subtraction". You will almost always find that your initial interfaces to the API are too clunky. Prune the interface down to its core purpose. The example he used was Contextual::Return. Context detection code in Perl is quite inelegant. You usually have something like the following: if (wantarray()) { return @list; } elsif (defined wantarray ()) { return $scalar; } elsif (!defined wantarray ()) { warn "why are you calling me"; return; } else { die "impossible, but expect the unexpected"; } An interface that produces more maintainable and readable code would be: use Contextual::Return; if (LIST) { return @list; } if (SCALAR) { return $scalar; } if (VOID) { warn "why are you calling me"; return; } else { die "impossible, but expect the unexpected"; } He went even farther and moved it all into a single return statement like: use Contextual::Return; return ( LIST { return @list; } SCALAR { return $scalar; } VOID { warn "why are you calling me"; return; } DEFAULT { die "impossible, but expect the unexpected"; } ); Next he explained in some detail how he implemented this using Want.pm and 'package DB' (as a proxy) and it went completely over my head, i.e., he lost me. So it goes. Another good way to 'Evolve By Subtraction' is to let the users (really testers) help. The module IO::Prompt evolved this way. Users will quickly tell you what they don't like about an interface. If they report about having to repeatedly do something whenever they use the it -- like turning on an option -- this is a strong signal that the option should be on by default. The fourth principle is 'Declarative Beats Imperative', by this he means enable the user to set options that describe what is to be done, don't make them do it. For example, the Exporter module, used to set how routines in a package are exported requires a lot of coding from the developer. Usually the code looks something like: use base 'Exporter'; sub import { .... } our @EXPORT = qw ( routine1 ); our @EXPORT_OK = qw ( routine2 ); our @EXPORT_TAGS = qw ( ALL => [qw (routine1 routine2)], SET => [qw (routine2)], DEFAULT => [qw (routine1)] ); Wouldn't it be nicer to just mark the subroutines in the package that are to be exported like this: use Perl6::Export::Attrs; sub IMPORT { my $pkg = shift @_; .... } sub routine1 :Export ( :DEFAULT :ALL ) { .... } sub routine2 :Export ( :ALL :SET ) { ... } Now it is easy to see how, if at all, a routine is exported, without having to check any of the EXPORT lists. The module Perl6::Export::Attrs enables just this feature. The fifth principle is 'Preserve the Metadata', or "Preserve the Users Metadata". Basically don't make the user re-enter data. His example for this principle was Config::Std module for reading and writing simple configuration files. All the previous configuration file modules would read the options in the file into a hash and write the file back out in the hash order of the keys, losing the structure of the file and any comments in the file. Not very nice. Config::Std keeps the comments by caching a copy of the file and unity this copy and any values modified, added, or deleted in the hash before writing the modified copy back to a file. The sixth principle is 'Build on the Familiar'. For example, consider Log::StdLog. Instead of using an object oriented interface for logging why not just create STDLOG. Most users are already familiar with STDIN, STDOUT, and STDOUT. So STDLOG is just file handle for logging. He did not that this package is not thread safe -- yet. The seventh principle is 'The Best Code is No Code At All', that is, just by including 'use PackageName;' you start using the API without having to write explicit API calls. 'use strict;' and 'use diagnostics;' are good examples of this. Another is 'Object::Dumper' package. Ever forget and print an object like: my $obj = myClass->new (); print $obj; Unless you have overloaded the stringification function you get something like 'myClass=HASH[0x98392819]'. Wouldn't it be nice if it just automatically did 'print Data::Dumper->dumper($obj);' for you. Well Object::Dumper does just that. He concluded with some good advice on developing an API -- follow 'The Way of the Camel". Have the API do the hard work for you without getting in your way. Really find the best defaults for the subroutines and work hard to keep the options small. Make the subroutines do one thing really well, maybe even better, overload a built-in function or operator. The API interface should be intuitive to use, if you can't remember how to use it then you need to improve it.

ETech 2006: ...Better Collaborative Filtering

2006-03-20T22:44:00.000-05:00

The talk by Charles Armstrong (an ethnographer) and CEO of Trampoline Systems was quite interesting. He studied under sociologist Lord Young of Dartington and at the School for Social Entrepreneurs (SSE). While studying at SSE he spent a year on the Isles of Scilly (photos), where he helped people learn computer skills. It was here that he studied the flow of information (or news) within the community. What was intriguing was how efficiently information valuable to a specific person or group was passed on to them as it came to the island (can't remember which island he was on, I think the population was about 100). Usually a boat would come to the island daily delivering supplies and information or news would be passed to the people helping unload it. For example, maybe the boat would not be coming tomorrow so the news would pass to everyone who planned to take the boat to the mainland the next day - well before the end of the day. His talked really had me thinking about Watts and Newmans paper Identity and Search in Social Networks. I think Watts's description in his book Six Degrees: The Science of a Connected Age somewhat models what Armstrong observed. His company, Trampoline Systems, provides "a technology that helps groups of people connect, collaborate and manage large quantities of information." From the presentation I concluded that the system would certainly provide an excellent means to do social network analysis of a company via its emails. I believe the system involves configuring your current servers to send copies of emails to a "database". From the Tos and Froms a network can be built, as in the photo on the left. (I'm not positive about the last three sentences.) Some type of content analysis of the emails is under development (maybe some type of automatic categorization) to facilitate the sharing of information. I didn't get how the this part of the system will work. I talked to him later in the day about the system but unfortunately we did not have time to go into any details. To get a non-technical understanding of good verses bad social or project networks within a company I would recommend The Hidden Power of Social Networks: Understanding How Work Really Gets Done in Organizations by Robert L. Cross, et al...

Sony Bravia Commercial

2006-03-17T17:31:00.000-05:00

Both photos are by sepiatone in his Sony Commercial flickr set.

I know this is old news but this is such a cool commercial. I so wish I could have been there in person.

ETech 2006: Future of Interfaces Is Multi-Touch

2006-03-15T20:51:00.000-05:00

Jeff Han's demo of his multi-touch interface was outstanding and pretty much stole the show for the day (March 7). The applications are limitless and could open up a whole new facet of user interfaces. Further, Han commented that the hardware technology was inexpensive. To appreciate the technology you really have to see it. There is a demo video on Han's web site and here is a video of his demo at ETech from YouTube. Take note in the demo videos how he uses hand gestures to change the angle of view of 3D images. Photo courtesy of James Duncan Davidson/O'Reilly Media.

ETech 2006 Session: Scaling Fast and Cheap

2006-03-14T22:45:00.000-05:00

Introduction Cal Henderson's all day talk on "Scaling Fast and Cheap - How We Built Flickr" is packed with information about building a scalable enterprize web application using predominantly open source software, some custom software, and very little commercial software. Anyone from a beginner to an experienced web applications developer would find the talk useful. Especially since Cal was open to any questions. However, if you want a talk on the history of Flickr this is not the talk for you. Currently flickr runs on about 200 servers and has about 600 terabytes of storage, not including redundancy it is about 200 terabytes. There are two data centers, one in Texas and one in Virginia. The development team is about 10 engineers and operational support is now handled by Yahoo's operations team, but previously it was about 4 people. Keep it simple He started out his talked with a picture of the godfather of computer science, Donald Knuth and his famous quote "premature optimization is the root of all evil." (Misattributed quote, see comments). The point he drove home was not to waste man power optimizing the small stuff or the stuff that won't need optimizing. Almost all optimization will be in the database and hardware configuration of your disk storage system. In that vain he recommended buying commodity hardware and install the standard version of linux, with one exception, use the compiled binaries of MySQL from www.mysql.com -- don't try to compile your own. Further, run mysql on a 64 bit machine to get around the memory limits of 32 bit machines. Don't lock yourself into a hardware platform if you can help it -- especially if you are growing fast. If you do, check that the components (disk drives, network cards, etc...) your vendor uses stay compatible with your version of linux. If you can't do that, then plan ahead for some time to develop for a possibly new version of linux. The main software components of flickr are Linux (I think RedHat, Debian, SuSi), JVM, Smarty, Mysql, Apache2, PHP4. Consistency across your systems is key to ensuring ease of maintainance and ease of development. Use Version Control for Everything Use version source control (CVS or Subversion) for everything, -- and as hard as it is -- put useful comments in the version control system. I personally like to recall a quote from (I think) Daiman Conway "Document your code as if a homicidial maniac who knows where you live will be taking over development of your code." To help with this use a simple CVS or Subversion program on the client side -- let your developers use which ever one they like. Also, put everything into the version control system, application code, system configuration files (apache, php, etc...), documentation, etc... Set standards for naming files, database tables function and object names, etc... Don't worry about settling on the perfect one, the greatest benefit is in everyone using the same standard. Use a bug tracking system like FogBugz, Mantis, RT, or Bugzilla. Get disciplined and fix bugs before doing development on the next release. Fix the easy bugs first (low hanging fruit). Categorized your bugs, P1 the production site is down, P2, causes the staging site to go down, P3, does not bring the site down but really needs to be fixed to maximize the user experience, P4, it is a bug and no one will every notice it. Local, Development, Staging, Production Application development occurs in four segments, local, development, staging, production. Local is the developers local machine where they have the component of the system they are developing on installed. Development is the developing version of the site, the lowest level that all the components are together. Staging is the almost live version of the site -- the in house test site. Production is the live site and is only updated via the staging site via a very simple interface -- one button click. So scripts are written to deploy the latest version of the staging code to the production site. Hence, it is easy to roll back to a pervious version of the code on the production server from the staging server. This is important because you will never ever be able to fully test a web application. There will always been bugs that can only be revealed on the live site. Unicode Flickr supports unicode and it is easy to support this in a web application. The hard part is data integrity. What do we do with invalid unicode? First, set up a data intregity policy for the site. Flickr filters the data (comments, titles, etc...) before it is stored to ensure it is valid unicode. HTML and SQL Spoofing Displaying user entered HTML on your site is a really stupid idea from a security point of view since there are so many spoofing attacks. However, if you are going to do this use an open source library like lib_filter to clean up the input first. SQL injection attacks are another problem. Use just-in-time escaping of the SQL input and never grant more permissions for a database user than necessary. Email Even with all the RFC's (561, 822, 1521) for an email standard a lot of email still does not adhere to these, especially mobile phone providers. Flickr uses PEARS's Mail:mimeDecode, iconv, and custom code to parse email. When you send a photo to your flickr account via a cell phone a lot of code goes into getting the title, comment, and photo out of the email. Some providers seem to take advantage of this feature to advertise via by replacing the subject line of the email with an advertisement or including icons at the bottom of the email. BottleNecks Bottlenecks almost always occur either because of swapping to disk because of high memory usage or non-optimized database queries. While it is possible you could have a CPU bottle neck if your web application requires some heavy crunching (image or video editing), it is usually very rare. Don't look for a bottleneck until it occurs. Remember, pre-optimization if the root of all evil. Try to build in some stats gathering into your site to help locate any bottlenecks, e.g., add a feature to optionally log the time of queries or other processes. Mysql indexes are a "bizarre black magic", when it comes to optimizing them consider hiring a Mysql expert. Site Monitor When it comes to monitor the site Cal was very happy with Ganglia for gathering all kinds of trend (past hour, day, week, month, year) stats and Nagios for real-time health (is everything up and running as it should be) monitoring of the site. Scalability Cal defined scalability as horizontal scaling; buying more servers not buying bigger servers. He centered mostly on how to get MySQL to scale. He describe the different backends to MySQL, MyISAM, BDB, InnoDB, and Heap, and the pros and cons of each and how best to use them. The most interesting part was their use of MySQL replication, which does not get you true horizontal scaling. It gives you the capability to handle a lot more reads to the databases but every database receives the same number of writes. Most web applications do far more reads than writes to a database. I won't get into the details of their Master/Slave database setup. One drawback to this is replication-lag – which you can experience on flickr with the tags. If you add a tag to a photo on flickr and then search for photos with that tag a few seconds latter the photo may not be returned. The problem is that your search request was handled by one of the database slaves that was not updated yet. True scalability somes with MySQL5 and Orcale RAC ($25K per processor). MySQL5 is not ready for prime time. So the only option is to use Orcale to write your own code to do this – which gets really complicated fast. Not recommended unless you really know what you are doing and have simple database SELECT's. Currently Flickr's architecture does not scale horizontally. They just keep stretching their current architecture. Storage I got the impression from the last part of Cal's talk that the image storage system is really not too complicated. He didn't get into the details of what hardware systems they are using but it is grown by just adding more hardware (scales horizontally). The location of the images on the system is stored in the database. I think they have written their own code or protocol to fetch the image from the storage system instead of using ftp or nfs – for speed mostly likely. He ran out of time before he could talk about RSS support and the flickr API. Building Scalable Web Sites by Cal Henderson will be available soon.