While working through my backlog of reading, I came across the results of Software Test & Performance magazine’s Testers Choice Awards for 2008. This list is of no use to me.

First, look at most of the winners – LoadRunner, TestDirector, components of IBM Rational suite and other high priced, commercial products. Working at a startup now (and several in the past), there is rarely budget for these tools. Oftentimes, the tools imply a process or a level of maturity that do not exist in the first couple of years of a startup’s existence. A more useful breakdown would be a set of testers choice awards for open source tools and a set of testers choice awards for commercial tools. (There is a category for free test/performance tool but that is about as broad as giving an award for best vertebrate – there is just too much variation.)

Which is actually the second point, because of the diversity of the software development space, a generic list such as this does not shed light on tools that may be applicable to my particular area. This is similar to an example James Surowiecki used in The Wisdom of Crowds regarding the generic top 10 lists in music. There may be only 1 or no songs in the top 10 list for the genre I am interested in and most likely I am already familiar with the song. If I am interested in that genre, the generic top 10 list only tangentially touches my interests and does not supply any new information. If I am looking for new songs like the one I like in the generic list, the other entries may not help. However, a top 10 list of songs in the specific music type (the more specific the better) may lead me to new songs which I will enjoy. The same is true when it comes to tools. A “testers choice award for java code coverage tools” might point me to tools I am not familiar with while the category “java test/performance” does not point me to any tools that I am not already familiar. Knowing which code coverage tool for java others consider the best is much more useful than knowing LoadRunner won the java test/performance category for 2008.