Before I begin, let me just say that this article was inspired by Andy Brice’s article The software awards scam. If you haven’t already read it, you definitely should. While I’m talking about Andy’s blog (Successful Software), let me just quickly add that I’ve been following it for some time and it’s very impressive. He has many really good articles, of which I’ve linked several across multiple posts here on this blog.

Getting back to the article, for those of you who aren’t familiar with software awards, many shareware directories (sites that are basically lists of software applications) sometimes give out software awards. What are software awards? The concept is that the better your software is the better the award you get. Maybe it’s a 4/5 stars, maybe 5/5 stars. Tucows (one of larger download sites) is famous for it’s cow awards (1-5 cows per software). Basically they exist to evaluate the software they list to let you know which ones are good and which aren’t.

Now if you think about it, how can some of these sites properly evaluate all the different types of software applications? They can’t possibly. What features make for a good property management software application? What features and usability make for a great mp3 player? What about a speaker testing software application? The list goes on. It’s possible but not on the budget many of these sites have. Especially when you consider that most are one person shops.

Even Tucows rating is very inaccurate. When LandlordMax was initially launched we paid for a Tucows expedited review (or whatever it was called) to get a detailed review report. We scored an average review. Not because of the features we offered or the usability, but because we didn’t provide some arbitrary features. These include:

• Flash-style or some form of professional visual intensive tutorial that aids in rapid learning of the program’s basic operation. (2 points). This is a nicety and doesn’t really tell you how effective the software is.
• Learning curve: “Does your application package offer quick launch, desktop or additional shortcuts?”. In no way does this describe the quality of the application.
• Repair function in the uninstall. How many applications really do have a repair feature that actually works?
• Functionality: “This is the Reviewer’s overall opinion of the functionality of your application. This criterion is rated in terms of functionality, speed, and resources.” Completely subjective. For example Photoshop sure takes a lot more resources than NotePad, therefore NotePad must be a much better application.
• Does it offer tips on startup? If it does, would that make the application any better? Would it help you play mp3’s with better sound quality? Would it help you balance your check book?
• “Linking customer support information into the application can gain you one to four points instantly, depending on what type of customer support you provide.” That’s for a whopping 9% of your total score! All I can say is wow. How can providing an embedded link to your customer service in the software make it better or worse, and especially by almost 9%
• File size. This is part of the “Program Enhancement” section. An amazing 3 points here. “If a competitor is offering the same exact features as your application and it’s half the size, it will receive more points that your application.” How can you accurately figure this out without a lot of effort, and even then… Unless you truly understand the benefits of Photoshop, how can you compare to the Windows Paint program?
• “Author home page: 2 points = The site contains contact information, brief program information, and online help and documentation.” Again, I agree that it’s great to have but it still doesn’t really tell me how good or bad the software is.
• “Cost vs. Value: With respect to the price you’ve set for your application, the reviewer’s ask themselves a question depending on what license type you fit under: (3 points)”. I’m sorry, but for property management and real estate software, unless you’re directly involved in the field you can’t possibly answer this question with any real accuracy. I’m sure the same is true in many niche markets.
• And the list goes on…

Overall, as you can easily see, their evaluation criteria clearly doesn’t measure the quality of a software application. It does measure something, and it’s should be consistent. Well even here I beg to differ. When we ran LandlordMax a second time through the Tucows rating service we got a significantly different score. Either way it’s pretty easy to game this system. If you provide a software that has an amazing installer, lots of documentation (the quality is indifferent), that’s small in size, and so on. you will get a good score on Tucows. The quality of the application (features, easy of use, etc.) is only a secondary concern. And it’s easy to understand why, there’s just no way they can accurately rate all the software applications submitted to them on an ongoing basis. Just reviewing LandlordMax requires at a very minimum some understanding of the property management and real estate domains.

And even with this poor rating scheme, it’s still only a rating scheme. They try to abide by it and do offer something. As long as you’re aware of what they really measure, they mostly do what they claim (giving them the benefit of the doubt here even as I had a somewhat contradictory experience with LandlordMax). When it comes to other shareware sites, unfortunately most do fall short. Especially with one person operations. There’s now way for them to review every piece of software submitted. It’s not possible.

And this is where Andy’s article really sheds some light into this industry. It’s something many of us suspected but didn’t know for sure. At least not until now that is. Andy took the time to write a dummy application (he actually just changed a text file to an exe by changing the file extension name from .txt to .exe) and submitted it to 1033 shareware sites. Many gladly accepted his software. Even better, many gave him 5-star reviews with nice shinny award graphics. Of the 1033 sites he submitted it to, 218 sites listed it and 394 are pending. Of the 218 sites, “approximately 7% of the sites that listed the software emailed me that it had won an award (I don’t know how many have displayed it with an award, without informing me)”.

Although I personally had a very strong suspicion it was a sham, I didn’t know to what degree (or for sure it was). We initially started to display the awards here at LandlordMax but quickly stopped once we started to get the feeling that they weren’t necessarily based on the quality of our application. We have some, but nowhere near as many as many as we could.

Although we stopped adding new “awards” we haven’t removed the older ones. Why not? This is where the reality of business and ethics step in. If we remove them and our competitors leave theirs on, then we haven’t won any awards in comparison. This might seem trivial but those awards can turn into significant differentiators, and hence real revenue dollars. We did receive them in good faith and have displayed them in good faith, well at least until it was recently confirmed otherwise. Now that we know we are faced with the dilemma of whether or not to remove them.

They’re still technically legitimate, someone did give them to us. No we probably didn’t earn them if we believe Andy’s report, but unfortunately I know Andy is absolutely right (numbers don’t lie). If we remove them we will probably lose some revenue from our less web-savvy customers. They will see our competitors showing their many awards and we won’t have any to match because of principles. Again, you have to look at it from the non web-savvy customer’s perspective, not you guys. They don’t know that most of these awards are a sham. Unfortunately we now know that the odds are very good that most of our awards are not “real”. The other side is that principles don’t always put food on our plates.

In this case I personally believe that the right thing to do is remove the awards page. In the long run our customers will appreciate our stance. They will appreciate our honesty. And therefore we will take down our awards page. Actually we might simply convert it to a “Reviews” type of page, similar to what Andy’s done with his website. I don’t know when this will happen since we’re already more than busy enough with our current workload (and this is a lower priority task after all), but we will make the change at some point.

{democracy:3}

Discuss (2)