Chill Out Guys

Jason Calacanis was in a huge twit about the Comscore blog numbers I posted about on Tuesday.

And Jeff Jarvis took a pot shot at the "panel research" methodology that Comscore used to compile the data.

First, I’d like to suggest that everyone chill out about these numbers. 

They are the first attempt by a leading Internet measurement firm to measure the blog world.  As I said in my post:

There are some questionable stats in this report.  Gizmodo over
Engadget?  Defamer over Boing Boing?  I have my doubts about some of
the blog specific data …

Jarvis says that panel research is "messed up" and goes on to say:

This is the method used by Nielsen et al to measure TV and radio and
print readership — affecting billions of ad dollars — and it is and
always has been relative bullshit. That’s why advertisers buy it,
though — because it is relative, because they can compare this magazine
to that magazine on the same sheet. But it’s all based on a small and
only allegedly representative sample of people. It’s meaningless.

Jeff is so wrong about this that I find myself shaking my head on this one. 

First, the problem with panel research of old is small sample sizes.  The panels that have been used for decades in old media are almost always less than 100,000 people primarily because it is cost prohibitive to collect data from a larger panel.  Clearly that is way too small for accurate measurement.

But Comscore invented the concept of a "megapanel" in 1999 and is currently measuring over 2 million unique Internet users. That’s the beauty of the Internet, you can measure online at a scale unimaginable offline.  At that kind of scale, panel research is not only accurate, its amazingly accurate.  As far back as 2001, Comscore was able to predict a missed quarter for Amazon.com.  And the panel size and the technology have improved significantly since then.

The issue with panel research is you need time to develop the statistical algorithms that weight the panel data correctly before you scale the numbers.  And you need a very good dictionary of the domains you track.  These take time to nail down.  Clearly Comscore hasn’t nailed it yet, but they never said they had.

When viewed as a first attempt at measuring the blog world with a rigourous methodology, this report is impressive and as I said in my post:

But that’s where Comscore is really strong.  They’ll take the feedback
they get from this report and drill down and eventually they’ll nail it.

Now on to Jason.  When he stops hyperventilating about the flaws in the analysis of specific blogs, he should go read the front of the report.

Everything he’s bitching about was fully disclosed up front and the idea that Comscore can be "bought" is so laughable that it makes Jason look silly.

Jason gives Comscore some advice in his lastest riff on the report.  And some of it is good, like starting a blog.  So I’ll return the favor and give Jason some advice:

Chill out, stop yelling at people and throwing around crazy accusations
Calmly suggest Comscore some ways to improve the methodology
Share your server logs with them so they can figure out what’s wrong
Make friends not enemies

The blog world needs good numbers.  Comscore is going to give them to us.  We’ll just have to be patient while they figure all this stuff out.