Whose Numbers Are Right?

Billions of dollars are getting spent right now on capturing Internet audiences. But we can’t even measure them correctly. As Magid Abraham, CEO of Comscore – the leading Internet measurement firm (where I am an investor and board member) – points out in this article in Mediaweek, the panel based methodology that Comscore utilizes is not necessarily less accurate than server logs. Most people think that server logs cannot lie, but there are many reasons they are not any more accurate than third party measurement methodologies. Magid’s article explains the reasons for this and I suggest you read it if you are interested in the reasons why that is so.

But let’s look at something else. Many people rely on Alexa rankings. They are free, easily obtainable, and so they are used frequently. But they are often wrong. Why? Well for one, they can be gamed.

But in my opinion, the bigger problem is that Alexa’s panel of toolbar users skews heavily toward the techie crowd. So if your service appeals to geeks, it’s going to do better in Alexa than Comscore.

Here is an example. Delicious (del.icio.us) is a service well known to the readers of this blog. I love delicious and use it every day. But it skews toward the techie crowd.

Answers.com is a more mainstream service. The primary way people reach answers.com is via the "definition" link on Google. It’s a decidedly mainstream service.

I have been an investor in both services so I am partial to both of them.

Here is a comparison of delicious’ traffic and answers.com’s traffic on Comscore:


In Comscore’s panel, Answers.com has something like 10x the number of users that Delicious has.

But take a look at Alexa’s numbers for the same two services.


In Alexa’s panel, Delicious has more users than Answers.com.

Who is right? Well I’d put money on Comscore being more right than Alexa because I think they have a more "balanced" panel. But I would not bet on Comscore being 100% right and Alexa being 100% wrong.

As Magid says in his article:

As always, the devil
is in the details. When you scrutinize the details, the answer to why
these two measurements don’t always match up is…“it depends.”

So what do you do? I think triangulation is the best bet. Start with Comscore, take a look at Alexa, take a look at the server logs, and maybe another measurement firm on top of that if you can swing it. Nobody’s 100% right. I think Comscore’s the closest, but nobody’s perfect.