Yesterday I came across an article on The Register about the anti-virus software AVG. To quote The Register:

In late February, AVG paired its updated anti-virus engine with a real-time malware scanner that vets search engine results before you click on them. If you search Google, for instance, this LinkScanner automatically visits each address that turns up on Google’s results page.

This has two very large implications, especially when you consider that more than 20 million people have downloaded AVG. And as you can imagine, it has to do with the bandwidth.

Let’s take an example of what happens when you go to Google and search for “latest movies”. In the past you would go to Google’s search page, enter in your search, and get some results. Then when you clicked on one of those search results you would go to that page. Simple, and it’s what you would expect.

However if you have the latest version of AVG installed something else happens. You start the same way, go to Google, enter in your search term (“latest movies”) and click on the Search button. However here is where things change for the worse. AVG looks at the search results and behind the scenes starts to download each and every search result webpage. This is without you having to view or visit the webpage, it’s all happening behind the scenes.

Why is this bad? Because most ISPs (Internet Service Providers) limit how much bandwidth you can use per month. It might be very high for some people, but now imagine that for every search you do, you visit 10 webpages (the default number of search results on most search engines). Even if you don’t look at any of the search results. You’ve basically increased your bandwidth consumption by ten times for every Google search!!! And not just Google, for all search engines, including Yahoo, MSN, etc. Your bandwidth usage has just significantly increased, by multiples.

But it gets worse. What if the webpages AVG decides to check out behind the scenes aren’t just simple pages but rather webpages rich in media that include videos, images, and so on? You’re bandwidth will be consumed in no time. And what if you have more than one computer on your network? What if you have 2-3 computers in your home? If everyone is searching at the same time you can imagine that your network will get slower because everyone is trying to load multiple webpages at the same time. And don’t even get me started on corporate networks. I can’t imagine the increased load on a corporate network with 10-1000 users!!! For that reason alone I suspect corporations will stop using AVG, the bandwidth usage is just too expensive.

And right now this is only officially happening for search engine results, but what if one day they decide to continue down this path and do this for every webpage? I’d hate to see the bandwidth usage on that decision! Most websites have more than 10 links on them! The front page of this blog probably has closer to a 100 links than 10 links on it. So instead of 10 times as much bandwidth, you’re looking at 10-100 times! You’ll be eating up your bandwidth cap each month. And corporate networks will crawl to a halt.

But what about the websites themselves? The websites you visit might also significantly slow down. Let’s take an example using the assumption that AVG visited every link on every page to amplify the issue (even without this assumption, on some search terms the increase in traffic can be very significant). When You go to my company’s website LandlordMax, the landing page (the first page) has at least 20 links. The navigation alone is about 10 links. That means our servers now would have to be able to handle 20 times as much capacity to handle the same number of users (assuming they all used AVG). That is, every person who comes to our website would not just download the first webpage, but instead they would download 20 webpages. And each time they clicked on a link, they would get another 20+ pages. We’d have to increase our server capacity by 20 times. That’s very expensive, and where do you think that cost would have to eventually be offset? Into the price.

But assuming it doesn’t get to this level, right now for every search term we get listed on the first page, we’ll basically get hit with a webpage request. So for example, if the search term “latest movies” get 10,000 searches a month (from people using AVG), and we’re listed anywhere on the first page, we’ll have to handle 10,000 webpage requests even though only a percentage of that search traffic will come to our site as shown in the above graph. Who’s going to pay for all that bandwidth? It’s certainly not AVG! It’s both you the user of AVG and the website owner.

It gets even more interesting, assuming bandwidth is free. In the example above, let’s say it’s now your website and you’re the 10th search result listing, at the very bottom of the first page. In this case you can expect to get about 3% of the traffic, or using our example of 10,000, 300 visitors a month. Now, with the new AVG system, you can expect this number to dramatically increase. Instead of just 3% or 300 visitors a month, you’ll get 100% of the traffic, a full 10,000 visitors a month. That will greatly skew your web metrics and webserver needs. In this example you’ve just increased your traffic (and hence server capacity needs) by almost 3 orders of magnitude! This will affect how you run your website/business. If nothing else, it will increase your costs which means it will have to be offset somehow (usually an increase in price for the customer).

Above this, a lot of the decisions for online businesses are derived from the web metrics. With this new system the metrics of your visitors are now completely useless. The 10,000 visitors a month mean nothing. They don’t represent the true scale of traffic. You’ve basically lost the ability to determine what works and what doesn’t. Although I won’t get into the details here, let’s just say that this means you can no longer correctly determine when and why you have a “real” increase in traffic. It’s all masked in a barrage of fake traffic. And because your web metrics are useless, your marketing now becomes guesses in the dark. Which then means more dollars have to be spent to make the same amount of money. Again this will eventually have to be offset into the price of the products/services on your website.

In any case, this appears to be a very good example of a lose-lose scenario. No one wins. Of course it’s not as drastic as I’m making it out to be, AVG only accounts for at most 20 million users. But it’s very important to realize that this is a significant amount of users. It’s enough that not only have consumers begun to take notice, but many websites are also starting to report significant increases in traffic and bandwidth due to this issue.

What’s really irritating about all this is AVG’s Chief of Research (Thompson) response according to the Register:

And if that causes problems for webmasters, Thompson says, so be it. “I don’t want to sound flip about this, but if you want to make omelets, you have to break some eggs.”

Be careful of the implications of your decision. This may result in a very large community backlash, especially when it’s backed by a large number of webmasters. We all remember the Intel processor fiasco of 1993, commonly referred to as one of the biggest technical blunders of all time.

And on that note I’ll leave you with some after-thoughts to ponder.

• What happens if the search result returns another search result (this could be maliciously implemented)?
• You’re now exposed for exploits from every website on the search results, not just the ones you visit.
• What’s your monhtly bandwidth cap?
• What will be the performance impacts on your machine for continually scanning all those extra webpages for viruses, spyware, and malware?

Discuss (6)