Three Statistics That Lie

I love the line about lies; "there are lies, damned lies, and statistics". You can use numbers to tell any story you want.

In the realm of web statistics, there are three numbers that are great to use if you want to tell lies. They are:

– RSS subscriber numbers
– Facebook app install numbers
– Follower numbers on Twitter, Friendfeed, Tumblr, or some other social media service

I tell you this because there is a discussion brewing on Techmeme this morning about how to get a lot of followers on FriendFeed. And I am telling you that the number of followers you have may be relevant early on in the life of a service, but it really doesn’t matter in the long run.

Let’s start with RSS subscribers. This blog has, according to FeedBurner, 133,000 RSS subscribers. That’s a big number. But the number of people who read this blog via the feed every day averages less than 4,000. Why is that? Well for one, that subscriber number has grown every day and never goes down. It includes people who stopped reading a long time ago, people who subscribed in multiple readers but now only use one reader, people who read once a month or once a year, etc, etc. Bottom line is the 133,000 number is basically useless.

Now lets look at Facebook app installs. Let’s look at the Social Gaming Network (SGN). Their Facebook apps have been installed almost 46 million times. And yet all the games together average about 650,000 daily users. Why is that? Well for mostly the same reasons. Some people install Free Gifts once, but rarely use it after that. I don’t mean to pick on SGN. It’s true for every Facebook app company. Our portfolio company Zynga has had 51 million of their apps installed and averages 1.7mm daily users. That’s the way it is in the Facebook app economy.

And the same is true with followers on Twitter, FriendFeed, and Tumblr. I have 5,152 followers on Twitter, 4,482 followers on FriendFeed, and  877 followers on Tumblr. I don’t know of any way to calculate the number of people who actually visit my updates on Twitter or FriendFeed, but I do know that my tumblog gets on average 250 visits per day. I suspect the radio of daily users/viewers to followers on Twitter and FriendFeed are much lower than Tumblr, maybe approaching the 3% number of feed readers to feed subscribers number.

Social media is no different from all media. The number of people who at one point were interested in your content or service is not that meaningful. What matters is the number of people who engage with your content or service on a daily basis and how engaged they are. And RSS subscribers, Facebook app installs, and follower numbers don’t measure that.

#VC & Technology

Comments (Archived):

  1. A3Munier

    on the spot!

  2. AndyBeard

    That attention from RSS is actually very much on the low side, I would have expected even for a more mature blog at least 10% especially with the way Feedburner seems to spike on post attention when someone shares a post on Google reader.

  3. Scobleizer

    Absolutely true! But that probably explains why I spend so much more time over on FriendFeed than anywhere else lately. The most engaging people are hanging out there. Twitter is a great place to do something other than engage with a lot of people. I’m still there too, but less and less, especially when it’s down.

  4. fascinated

    Imagine then, the people who start doing anything specifically to grow these bogus numbers! Creating value vs numbers ๐Ÿ™‚

  5. gregorylent

    which is why calculating return on advertising spend will increasingly be a crap shoot .. (my father’s phrase)

  6. NICCAI

    It would be nice to know which tweeters actually have a pulse! I’d venture to say that 70%+ of mine are bots.

  7. Joshua March

    Good comments. However I would just point out that Daily Active Users on Facebook is that – daily. The actual number of Active users is likely to be much higher (e.g. active once a week, or once a month). These numbers are just being introduced by Facebook, although currently it’s only the developers who can see them.

  8. Chris Phenner

    While no doubt there are huge discrepancies between “Initial Interest Indications” (III’s, perhaps) and “Ultimate Visit Indications” (UVI’s, perhaps), a ratio that divides the latter into the former might be a useful metric of health. I would envision a slide (to the board or to investors) that showed a “comparables” chart of similar apps/services that showed the relative ratios of UVI’s as a percentage of III’s.To inflate a service’s “value” using only III’s is indeed a “lie,” but to discredit these factors as “useless” seems to discourage more creative (or retention-driving) efforts that leverage RSS, FB and/or Twitter/etc. If I were running or measuring a “Retention Marketing” team (which is a hard job), I would have one component of that team’s bonus tied to the improvement of the aforementioned ratios.

    1. fredwilson

      The daily actives/total subs number is interestingBut it tends to go down over timeMy feed has been out there for almost five yearsIf I had started blogging six months ago, the ratio would be much higherfred

  9. moon

    Looking at the conversations on Friendfeed and there mostly about Friendfeed

  10. SexySEO

    Figures by themselves don’t mean anything: figures are just figures. And statistics were (and always are) the way of …. manipulation of figures :)))

    1. Leonard Boord

      Not so quick here… I agree RSS and Followers. But…Apps: When a user downloads and app he leaves the door open for more inbound communications. You can now devise different strategies to recoup the user, at a very low cost. So it’s not worthless, or meaningless your foot is in the door.

  11. seemsArtless

    But aren’t the statistics useful when comparing one blog to another? It is valid to say that a blog with 133,000 RSS subscribers is twice as popular as one with 66,500 subscribers. Sure, you’d have to keep some factors in mind, like how long the blog has been around, but I think the comparison is relevant and useful.

    1. fredwilson

      I don’t think it’s valid to say that a blog with 133k RSS subs is twice aspopular as a blog with 66.5k RSS subs.I think you need to look at average daily readers (web+feed) to see which ismore popularIt’s entirely possible that a blog with 66.5k RSS subs has more averagedaily readers than this blogfred

  12. Christopher Finke

    It sounds like FeedBurner needs to re-think how they calculate RSS subscribers – anyone who hasn’t accessed the feed in over a month definitely should not be counted as a subscriber.

  13. davidcushman

    amen Fred. The truly valuable are those who share your purpose and are responding to you right now.

  14. Jared M. Spool

    I think you’re right. I suspect that subscribers and followers are the 2008 version of 1999’s page hits.Great post.

  15. Dario Salvelli

    I think that is important also how people spent time to follow. It’s an important number, above all for the RSS subscribers.

  16. Ken Yarmosh

    A technicality here, although I agree with your overall premise. Here is how FeedBurner (Google) describes “Feed Subscribers” for a single day subscriber total:”FeedBurnerโ€™s subscriber count is based on an approximation of how many times your feed has been requested in a 24-hour period.”The key point is the 24-hour period. You will probably notice that the RSS subscriber one day count still drops on the weekend.So, while I agree that reach is more important than subscribers, I think it is a bit much to say that the subscriber count is “basically useless.” You can also look at your subscriber count over 7 / 30 days / all time, which then shows average subscribers during that time period.

    1. fredwilson

      Yeah, but a lot of those requests are from software, not users

  17. chudson

    I don’t think I would say these statistics lie. I think they’re just easy to misinterpret. In the earlier days of the web, people got really excited about pageviews. Then they realized there were more meaningful ways to track audience engagement (7-day actives, 30-day actives, unique users per month, etc) – crude tools like pageviews and installs are only good for first-order guesses as to how well a service is doing. I’d argue the statistics you cite (installs, followers, etc) allow lazy people to stay lazy if they choose to rely solely on those without asking more questions about the underlying level of activity.

  18. centernetworks

    Great post Fred – the only thing that I would disagree with is that the numbers don’t matter in the long run. I wish they didn’t but from what I can tell, advertisers sometimes make decisions based on these “lies” – especially the RSS one for bloggers. Your analysis though is spot on. One additional note, just like I wrote about Friendfeed defaults, there are many RSS apps that provide a set of defaults – that provides probably the highest boost for some blogs.I wrote about this a year ago on a post titled “are rss subscribers equal to hits” – rss subs as a metric reminds me of 1995 and the usage of “hits” as the publicly reported web metric.http://www.centernetworks.c

  19. kidmercury

    IMO some better metrics:% of traffic that comes from type-ins & bookmarksviral metrics — i.e. % of posts that go viralaverage time on siteavg # of non-spam commentspercentage of posts made by passionate users (i.e. repeat commenters)though ultimately i personally am not a huge metrics person and favor more intuitive stuff like feeling out which communities have the most passionate users.

  20. Blogwatch

    Didn’t FriendFeed add Robert Scoble (and a few other “A listers”) to EVERYONES Friends list? Basically counting on people’s laziness in not de-friending them, similar to “Tom” being everyone’s default friend on MySpace?

    1. Scott

      totally – this definitely distorts the metrics as well

  21. Ross Mayfield

    Sounds like you are about to discount my cashflows

    1. fredwilson

      ๐Ÿ˜‰

  22. James Joyner

    Another thing that distorts RSS subscriber counts is automatic includes. Several of us are on various “recommended” lists put out by various readers and RSS interactive products, totally inflating our numbers despite, as you say, many of those people never reading, let alone engaging with, the blog.

  23. Kyle

    Now that we know what doesn’t matter – what are the 3 web statistics that “don’t lie”? Or said another way, what are the 3 most important web statistics that my social media site should rely on?

    1. fredwilson

      Daily active unique visitorsAverage length of stay% of return visitorsThose would be my votes

      1. jer979

        Kyle beat me to the point on the question, but looking at a data point of one (me) and my daily interaction with you.I subscribe via RSS and read your posts pretty much daily, but I don’t always go to your site. Some days I don’t comment at all, today I’m commenting via disqus only. In all scenarios, I’m “engaged” w/the AVC brand.So, while those three metrics are certainly important, it relies on the website as the destination. That flies in the face of the distributed web/syndication value prop.You’ve posted before on the value of re-purposing and syndicating your content. There’s got to be some metric for that.

  24. Jesse Farmer

    Hey Fred,I’m Jesse, the creator of Adonomics, although I’m no longer working for the company that now owns it. A few points.1. DAU is not a very honest metric, either. It includes both new users and returning users. If your app is growing virally the “activity” can be fueled almost entirely by the acquisition of new users who do nothing more than install the app, invite their friends, and leave.On top of that DAU obscures some user behavior. For example, if I get 1,000 DAU today and 1,000 DAU tomorrow, how many users do those two days have in common? Is it the same 1,000 users on both days? 800? Or did only 100 users visit both days?2. Installs are useful, but not for what people think โ€” they measure the application’s network saturation. An application with 30MM installs has saturated almost 50% of Facebook. The growth characteristics and considerations of an application in this state are much different than an application that hasn’t already made the rounds.Also, for what it’s worth, the “install” numbers on Adonomics are off-the-wall wrong. It calculates them by dividing DAU by %DAU. Facebook doesn’t care much about the accuracy of these numbers so you can get day-by-day swings of over 2-3MM installs. Trust me, no app gains or loses that many installs in a day.That said, the way people blindly follow other people’s metrics is a problem. Metrics should support business goals, not the other way around. I think people see it as a game: get the “most” installs, get the “most” followers, get the “most” pageviews, etc.Just some food for thought.Cheers,Jesse

    1. fredwilson

      Jesse ยญ thanks for stopping by and adding to the discussionThis is terrific feedback and information

  25. ProductArchitect

    But the number of people who read this blog via the feed every day averages less than 4,000.Do you have any details behind that number? I’m very interested in understanding where it comes from.Scott

    1. fredwilson

      Its the feedburner daily reach stat for this blog’s feed

  26. sent2null

    nailed it, good article.

  27. spira

    I hate the phrase “”there are lies, damned lies, and statistics” – it’s nonsense. “There are lies, damned lies, and words” is just as meaningful. Statistics, by definition, can’t lie – they just are, just like words. It’s people who lie.Statistical illiteracy is a more widespread problem in our society than language illiteracy, but that’s the fault of our educational systems that think it’s far more important to teach high schoolers trigonometry than statistics. But that just means that more people can be fooled by statistics; it has nothing to do with the intrinsic nature of statistics.So please try not to slander statistics in the future – they’re victims here too.And yes, I’m (mostly) serious)

    1. Boudicca

      SpiraSorry if you are a statistician and that phrase upsets you and I strongly agree that statistical illiteracy is a Huge problem in our society. But I have to disagree that the phrase in nonsense.The line isn’t “there are liars, damn liars and statisticians” : )Of course it is people who lie – lies are told by people,damned lies are told by people and its people who lie with statistics (sometimes intentionally and sometimes simply because of incompetence).

      1. fredwilson

        Thank you for the correctionAs usual the commenters have the right answer

  28. Jay Cuthrell

    This is also why 90% of the peer reviews of 50% of all studies show that 75% of most statistics are fabricated and the remaining 25% are mired in pseudoreplication.And yes, I’m (mostly) joking.

  29. Ken Yarmosh

    “Subscribers is not computed for browsers and bots that access your feed.”http://www.google.com/suppo…(btw, this was supposed to follow in response to the above comment but it got placed at the end of the thread)

  30. Hootan

    Fred, your post makes me think of web statistics in general, like monthly uniques. Entrepreneurs quote me “millions” all the time, but a quick check of alexa or quantcast reveals another story (I know those sites are rough estimators, but decent nevertheless).I would imagine that the “big” numbers quoted by some of these new companies are meant to excite investors. I wonder whether a small but loyal following would be as exciting to investors as a massive but “undefined” following. The truth is more important, but in this day and age, it look like the market rewards huge growth in numbers, not quality of visitors.

  31. Douglas Karr

    There are a few infamous bloggers who have exploited these vulnerabilities and become as large as their numbers said they previously were. I won’t mention names here, but they’ve even bragged about what they did. The result? Bigger and better advertisers than they’d ever imagined with 6-digit incomes.You and I may be aware of these, but advertisers are not.

  32. bfernald

    Statistics and metrics are NEVER meaningful with out context. It just doesn’t matter how “accurate” they are, they don’t have meaning until you put them to work. Batting averages mean nothing until you’re faced with choosing a pinch hitter. Gas mileage (mpg) doesn’t help if you don’t know what kind of driving you expect to do.Subscriber counts are like horsepower. It’s latent potential. If you’re buying a Prius, you probably don’t care until that one day you need everything it’s got to make a left turn in front of traffic. Everyday activity on your blog may trickle as you broadcast your thoughts, but when you ask that burning question, suddenly everyone turns out to comment.

    1. fredwilson

      Subscribers = horsepower. I like it

  33. Paul Edmondson

    One stat I’ve found interesting is a velocity rate. Take the number of subscribers of a feed or installs of an app and divide it by the number of days (time). That will give you a ratio that you can track over time. I use this for several things in our business. If you graph this over time, it can be telling.

    1. fredwilson

      Totally agree

    2. Dorian Benkoil

      Paul, what’s it tell you?

  34. Marcel LeBrun

    To use a high school science analogy: it is like the difference between potential energy and kinetic (or motion) energy. Someone with 5,152 followers on twitter has a greater potential for engagement than someone with 100 followers, but not the guarantee. The right way to measure engagement or influence is by measuring the actual conversational dynamics which truly indicate engagement (comments, votes, bookmarks, links, etc.) around a topic or post.I think your point is especially visible with services where users/followers/subscribers constantly accumulate (like the facebook installs – since users may never remove them, even though they are rarely used). Good post.

  35. John

    I would throw out the idea that what’s needed is more specific measurements for many of these. Take, for example, Twitter followers. The number is generally meaningless, unless you could determine what % of those followers have notifications turned on… and what’s more, if you could actually measure the number of impressions of folks who actually saw or were delivered your tweet. The sad part is… all of these numbers could be reported, they just aren’t!

  36. ajayinsead03

    I just stumbled at your blog. Although I agree with the general idea behind your ascertain: I dont agree that the numbers are completely useless or lies — misleading – perhaps. Perhaps, because it depends on where you want these numbers to LEAD you ?While many have given different analogies..How about an analogy, from the world of Finance – from options.. Real Options..When people install an app, or tune in – > its an option they excersize – are they not? If yes, then that option has to be worth something… is it not? Think of the economic value ( not the monetization ) that such an option entails: * X millions have downloaded this bar –> u could be the next ๐Ÿ™ *My 2 centsCheers,ajay

  37. Chase Barfield

    Mr. Wilson – Very insightful and so true.

  38. Vijay Goel, M.D.

    You’ve hit the lazy thing straight on. While I was at McKinsey, I did a ton of customer segment mining through existing company data trying to match business results with customer data. Its not that the data doesn’t exist in most cases, its that people don’t actually want to make the numbers mean something, because then they’ll have to get off their rears to fix the issues that emerge.It’s important to track what’s happening to individuals to really be able to understand what your customer base is doing (e.g., rapid subscribes may mask an unhealthy early attrition rate if you’re only monitoring overall retention).Its also important for the company to know which customers (and their activities) make them money and what causes that loyalty to disappear. Nothing like having customer service maximizing their handle time per average caller when you’re an account decisionmaker thinking about renewing your relationship with a company…

  39. Boudicca

    Love the opening line!The question of how to quantify the number of people actually interact in a meaningful way is a big question made up of numerous pieces. I really like one of the pieces you brought up here – what is the ratio of of users/viewers and what are the variable that contribute to the variance in ratio between different sites.

  40. Kalpesh Khivasara

    Hi Fred, couldn’t agree more with your point. BTW, can you let me know how did you”guess” that the people reading your feeds number less than 4000? I’d be very interested to know.

    1. fredwilson

      Its not a guessFeedburner reports daily reach numbersFred

  41. I DISAGREE STRONGLY.Two thumbs up for sexy seo commenter, I like your call sign/trademark. I am going to create my own called SEXY BEAST!—Anyways, back to the topic at hand. After reading all the comments, I have to disagree with you Fred.STATS do not lie at all, they are what they are. It is the cultural perception to take it for its face value that has become the stinker. We could go a step further and include more detailed stats should an inquiring mind explore. Maybe have the feedburner count turn into a link where you can get far more detail stats for the public like someother companies currently offer. I think sitemeter.com, I am not to sure of the top of my head.I think your argument though has good points and got me thinking about influence. One solution can be the one I describe above were details stats are always are available for the curious minds where it is simple and easy to access.This then got me thinking about how influential a person is within their domain. There stats are insight, but what it comes down to is how much influence do you truly have.I was just reading about that company earlier this week about how they are working with Meebo to identify how much influence a certain individual has for ad purposes. I think that is just brilliant that a company is finally pursuing this next step in marketing, BRILLIANT!Breaking it down to the person :-)Sorry for the rants, great post!

  42. AngelaE8654

    I am still just learning about all these social bookmarking places. LOLAngela

  43. Chris

    This makes me feel good since I’m not really active in Twitter and I just opened a Facebook account. Now at least I know that the people who visit my blog, albiet modestly small, are quality visitors. But don’t you have to drive a lot of traffic to your site in order to attract a few good ones?

  44. Melanie Baker

    Hi Fred – Thanks so much for this, and excellent timing! Served as the perfect example to illustrate some work we’ve been doing on measuring social engagement as opposed to just “storyless” numbers: Storytelling ROI: social engagement metrics for Marketing & Social Media bloggers (As a side note, your overall engagement score currently is 28919, very respectable.)I think sorting out all these metrics and providing relevant context for people is still in embryonic stages, and there is so much potential for really cool developments. At the same time, though, the true connections between people, the incredible value of what you can learn from others — I don’t think stuff like that will ever be accurately analyzed. Nor does it need to be. ๐Ÿ™‚

  45. Hashim Warren

    another number that lies – unique visitors and pageviews.What really matters is engagement. What really matters is loyal visitors.

  46. Sagar

    i’m a facebook developer and yes, facebook install numbers are HORRIBLE…hahah

  47. Dorian Benkoil

    One thing that bothers me a tad about this is it seems to assume a user is a user, period. That it’s all about aggregate numbers. That may be true in mass media (Desperate Housewives, the Super Bowl), but in a targeted medium, 3,000 of the “right” people may be more valuable than 3 million of the general public.

    1. fredwilson

      That is very true and I am particularly blessed with the quality of the people who read this blogAlthough I do run advertising on this blog and in the feed, which goes to charity, the real value is in the relationships it has allowed me to develop with the readers

      1. Dorian Benkoil

        Thanks for the nice words, Fred. I was talking in purely economic terms, as well.Just one example: For IT-oriented lead generation, each individual lead can be worth hundreds of dollars, and it’s better to have a few high-qualified leads than a mass of unqualified.

  48. Hannes Johnson

    Interesting points – and very true.But don’t forget that your 877 followers on Tumblr might only view your content on the Tumblr dashboard without going to your site/tumblelog – so you got probably a bit more than 250 people reading the content you post on Tumblr.

  49. Antman

    But they make real good marketing numbers . . . perception is reality and for many they create one hell of a perception.