Celebrating Aggregation

Lately, it's been all the rage to bash the services on the Internet that aggregate content. Robert Thompson, editor of the Wall Street Journal, said this about aggregators a few weeks ago:

[they are] parasites or tech tapeworms in the intestines of the internet

Since I read that, I've been calling every one of my favorite services on the web "tapeworms." It's a compliment of the highest order. Because I love aggregators and they are the only way I get to content anymore. I don't read the NY Times directly, I don't read the Wall Street Journal directly, I don't read any blogs directly, and I don't watch videos or listen to music directly. I've got dozens of aggregators that I use daily and they take me to the content I want to consume.

The Internet and digital media broadly has produced a glut of content. There is no way that anyone can consume all of the content that is available and relevant to them. And no media property, be it the Wall Street Journal or any other content creator, can produce even 5% of the best content I want to consume daily. These media companies used to be the distributors and creators of content. But now they are just the creators and they will never get back to a position of being a distributor unless they become "tapeworms" too. Watching them insult the best services on the Internet tells me that they aren't headed in that direction.

One of my favorite examples of aggregators is Tastespotting, one of the Gotham Gal's favorite web services. Here is the front page of Tastespotting right now:

Tastespotting is like Techmeme or the Hype Machine or Digg, but it is for recipes and the photos that accompany the recipes. If you click thru to Tastespotting, and you should, you'll see that these images all link out to blog posts by regular people. There are millions of people who cook, love recipes, love food, and many of them have taken to photographing their work and blogging about it. The Gotham Gal is one of them.

The food magazines and even the food sections of newspapers could have done what Tastespotting did, and they still can, and they should. Because there is a ton of great food content out there on the web and aggregating it up is more valuable to readers than trying to do it all yourself with your editorial team.

Aggregation is the central element of distributing content on the web. It's not going to get shut down by calling these services names, suing them, or even worse taking your content out of them. The best and only thing media companies can do is join the aggregation parade, celebrate it, and get good at it.

Reblog this post [with Zemanta]

Comments (Archived):

  1. nickhalstead

    Totally agree, they are scared by what they see – and would right now prefer to try and stop the aggregation. Let us hope they see the error in their thinking, but am sure AP in general and lots of the companies that make it up all think the same.You may also want add tweetmeme.com to your list Fred, we do have a Food & Drink section which picks up all kinds of stuff people are sharing on twitter -> http://tweetmeme.com/?categ

  2. Jason

    must say it in light of this discussion – same rule applied to internet business:”It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change.” ~ Charles Darwin

    1. fredwilson


  3. Jevon

    Aggregation and applying interesting filters to that aggregated content is one of the really unique things we (by that I mean,.. everyone) are experimenting with right now on the web. Blogs vs. Newspapers, etc isn’t such an interesting change, because it is arguably incremental. Blogs are cheaper newspapers in some ways, especially blogs that emulate newspapers.Aggregation however is not incremental and it is not derivative, it is new. That might not seem like a big deal to most people, but it is interesting to me because we don’t yet understand it’s full impact and what sort of market will grow up around it.That’s part of why I like what @howardlindzon is doing with stocktwits. If he was simply creating a place for people to write their thoughts about a stock, then they would not have been creating anything really new., what he did though was let people create that content elsewhere (it could be anywhere, it could be right on stocktwits as well) and instead focused on building a really smart aggregation and markup system, which is where the value really gets created.Push publishing models are getting boring, Pull aggregation models are where there is an opportunity to do really new, and potentially big, things. It’s where value gets bundled up. Network effects are no good if you can’t identify them and use them, aggregation is where you can do that.

    1. Jevon

      Another aggregation model that is working is in Travel. Aggregators like Kayak.com are adding a lot of value for users.It might not seem like an analogous concept to content/media aggregation like you are talking about, but there may be more parallels than are immediately obvious. The main one being that as many airlines and hotel companies can be aggregator to fulfill a travel need, many content creators, editors and curators may ultimately be needed to fulfill an information need.

      1. fredwilson

        ExactlyWe have a jobs aggregator in our portfolio called Indeed.com which is doing greatIt’s a lot like kayak

        1. guruvan

          As a job seeker, the endless jobs aggregators are just killing me. Seeing the same jobs posted over and over and over to all these sites just seems like a huge waste. Most of the jobs seem to have been posted on Craigslist (usually by the hiring company) in the first place. What’s the point of adding even more to this? I’ve found very little additional value in any of these, including the above mentioned one.(color me a little frustrated on this issue)What none of the jobs aggregators seem to (want to) do is add any additional value. If they did that then my tune would be different.Aggregating content without adding value to it just doesn’t seem so useful (in any field, not just jobs)

          1. fredwilson

            Well over 10mm people a month seem to be getting value from indeed

          2. guruvan

            lol..I knew you would say something like that. I was hoping you would see the point there though.I checked it out in the first place because I was seeing recomendations for it from everywhere. I certainly can’t ignore people recommending job sites. But when I got there, all I saw were the same things posted everywhere else, and I was at a loss to understand what the difference was between Indeed.com and everything else.I’m not saying that its not just fine, I’m just looking for something that a site does besides aggregate content.And the key word in your comment is “seem” How exactly can you tell if they ARE getting value from it? The recommendations give me some indication, but I think people are likely to recommend just about anything that’s new on the net.The REAL indicator of value of a job site in particular (to me at least), is “How many people are being hired from jobs they found on that site”I have to wonder if 10MM people aren’t using it with the thought “leave no stone unturned” in the hopes of finding the very few posts that they haven’t seen everywhere else. And of those 10MM people, how many are searching there 1,2,3,5 times a week? Just having that many registered users doesn’t mean that much.I’m not trying to denigrate the service, just trying to understand. I was hoping that as an investor you could shed some light on that.

          3. fredwilson

            All we have is anecdotal data and from that I can tell you people find and get jobs on indeed every day

    2. fredwilson

      Excellent pointHoward is a genius and I really want to see him start aggregating other sources beyond twitter

      1. Jevon

        Just be happy that Howard doesn’t change his name to Cramer and make stocktwits one big video blog.

  4. Nate

    The Wall Street Journal is itself an aggregator. They’re just mad that now we have better aggregators without their high fixed costs. Boo hoo.

    1. Jerry

      I couldn’t agree more. I’ll go further and speak as a former reporter…a huge percentage of “news” is derivative (and after all what is an aggregator but a more efficient and intellectually honest derivation?)…reporters follow each other’s leads all the time. Just think of the number of “stories” reported on broadcast news that are essentially rehashed versions of stories first published elsewhere. (And I won’t even mention the number of business “news” stories that began with overt or covert public relations spinning.)Thompson’s Holier Than Thou statement is crap. What’s behind his statement is fear; fear that his “style” of journalism, the printed paper, is becoming irrelevant. Long, long ago in a galaxy far, far away I was a print journalist who talked about the end of print causing not only the loss of the paper medium but an entirely new way to gather and publish information.Instead of worrying about aggregators, Thompson should focus his reporters on breaking original stories. The current lead on the paper’s website? A rehash of the swine flu epidemic with the news hook being that “officials are on alert.”I’d like to know if this is really the beginning of a new pandemic or yet another example of overhyped, overblown fear mongering and I’d like to know now.But maybe I’ll have to read an aggregator site, and piece together the information myself from a variety of sources.

      1. fredwilson

        Jerry – I just spent five mins trying to find a health news aggregator and I couldn’t find oneWebmd is missing a huge oppty herethis is the best I could come up withhttp://bit.ly/SLacLcheck out the related links at the end of the storyugh. We can do better.

        1. Shafqat

          I havent used it, but I believe RightHealth is also in the health news aggregation space, are are either #1 or #2 in the space based on traffic. It’s basically Kosmix, but with a different UI and only focused on health news.

          1. fredwilson

            ThanksI’ll check it out

        2. James

          Fred,http://www.yourhealthtopics… is a new health news aggregator. You can find the latest health news about dozens of different conditions and treatments. We just started it a couple of months ago, so it’s still very much a work in progress, but we’re getting there. Our approach is to use semantic text processing to understand what blog posts are about and aggregate them around topic areas. We also started other topic-specific aggregation sites around travel, sports, gossip, financial news, and tech.

          1. fredwilson

            Cool. I’ll check it out

  5. smikolay

    The benefits of aggregation are certainly becoming clear, as you say sites like hacker news or new mogul facilitate the discovery process. An interesting development, in my mind, is the advent of white label services that facilitate this kind of aggregation site (like slink set), which mean that anyone can create a niche aggregator.Having in the last few weeks felt a bit overrun by a lot of the green washing news that comes out on environmental topics, I slapped together an environmental aggregator in the hopes of bubbling up the better stories on the topic. Even with just a handful of people using it, my discovery process has changed and I’m finding things I otherwise wouldn’t have.I suppose the ease of creation raises the possibility of a downward spiral – where initially people were willing to read a dozen (or so) blogs everyday, but became overwhelmed when the reading list grew into the hundreds, they may now be willing to use a dozen or so aggregators, but could have a similar response if that number grows too large.Fred, as you say you use about dozen, I’d be curious to know what the other ones are.

    1. fredwilson

      I use techmeme, hacker news, tim o’reilly’s twitter links, dave winer’s 40 most recently links for tech newsI use new mogul, seeking alpha, and stocktwits for stock/business newsI use Huffpo for political newsTastespotting for food/recipesHypem and wearethehunted for musicAnd there are more, so its probably a couple dozen actually

  6. Jstake

    Aggregators are great. I first found tastespotting when it was part of the notcot sites. Curated content, like drinks with , or design with notcot.org, or even news with Slashdot.org, really helps sift through the endless piles of junk swirling around the net. I can’t wait to see it catch on even more.

  7. daryn

    hallelujah!There’s a huge value to context, so it is very important that these individual pieces of content continue to live in their original homes, but aggregators such as tastespotting add a tremendous value for discovery and breadth of community outside of the original audience.

  8. rickburnes

    An irony here is that traditional publishers bash the web for its supposed lack of serendipity (the daily me argument). Of course, it’s the aggregators, which they also bash, that do the most to help solve this problem.That said, a question for you: Do you think aggregation is a different (or better) business than content? Seems to me that it’s pretty much the same. You have infinite competition (everybody with a tumblelog, FB or Twitter account is an aggregator), so the best you can do is lead a niche. That’s what Digg does, that’s what Techmeme does, etc. Hard to imagine how either of those services grows beyond their niche.

    1. fredwilson

      Well at least the cost of operating an aggregator is minimalI think we are in a new era in media where success means a lot less revenue and a lot more profit

      1. rickburnes

        Interesting. I think you’re right about media success.And they corollary is that media mediocrity means a lot less revenue and a lot less profit — super news for those of us that consume media.

      2. kidmercury

        “I think we are in a new era in media where success means a lot less revenue and a lot more profit”damn boss try to give us a warning if you are going to drop massive doses of truth like that

        1. fredwilson

          I’ll tweet it first next time 🙂

  9. Tina White

    I’m a mom with two young kids and one area i’d love to see an aggregator is personal information. My expense management is on mint, my health records are all over the place, my kids info are on outlook, facebook and god knows where else. It is a security and management nightmare! My husband says that is how 90% of people’s info is – on the internet, on their PCs, on their phones and on paper files and sticky notes. Maybe i’m a worry wart but with the ability to do more on the web from paying bills online to buying everything it would be great if we had a personal life aggregator;)

  10. Jon Michael Miles

    I live and die each morning by http://www.popurls.com. I have another friend that loves Newsmap – http://marumushi.com/apps/n…Aggregation (and viral memes that inhabit them) both solidly bring up the issues faced by content providers. When we launched our website we had to fully embrace the idea that our content would most likely not be consumed on our site, and that’s a good thing. Now with a book and Kindle for that matter, this totally makes sense, content has a fixed price and is inherently portable. But we are also a “digital” content provider.Both for our atoms and bits content we use both push and pull – push via email, social bookmarking etc, pull via all the push, and ads via Facebook (very targeted but limited click through), radio, offline etc.But this doesn’t address or technologically solve the money piece for content with legs.I recently was on a panel called “Media Unhinged” for the Women in Film and Video chapter in DC, (it’s for video people on how to think about the web, you can get a PDF of my Keynote here – http://tdi-presentations.s3… Copyright 2009.pdf) and one of the other panelists Jen Sargent from Hitfix.com was there talking about her site. She had spent six years at Variety helping it’s transition from print to web, so she had a HUGE head start from an ad sales perspective. She knew exactly what advertisers wanted and is giving that exactly to them.But not all content in the world can be celebrity focused 18 to 34. Ours is not. But we still need to make money.If we as a content provider ever experience the dream oft chase after viral hit, the places to monetize, and methods of monetizing are still archaic. Google adwords? Metacafe like the Diet Coke Mento’s guys?So just to realistically monetize our online content we have to have offline content such as books. Even well known content companies like Jibjab have had to come up with other ways to get money in the door via e-cards and so forth.Layer on this, that I have friends in LA and New York that are producing web and mobile content simply as an audition for entry into mainstream content – hoping to build an audience and transition to commissioned programming. There again, having personally been on the inside of how TV, print and other types of content gets commissioned I’m here to tell you it all boils down to what the ad sales department thinks.This reflects a certain disparity where again individual content providers are working under the illusion that they need to make to the ‘big league’ whatever that means. And when they do, it’s still not a guarantee of anything.For example, Susan Boyle, from “Britain’s Got Talent” – Is she making money? It’s really the feeding frenzy around her that is doing well off her meme. She is easily at the center of many millions of dollars in ad revenue and huge associated PR heat, but I’d be surprised if she had made more that 150k on the deal at this point. So the system, no surprise, is not built to do anything except feed the online variant of offline traditional media.The recent moves by AP and consortium visa vie google ad revenue are a good example of how the revenue models need to be revamped, but it is too reminiscent of content providers fighting over cable revenue share.Unless I’m missing something there is still alot of work to be done, and a lot of opportunity to be made in terms of content providers INDIVIDUALLY making money off the aggregation and viral meme phenom.Jon

    1. fredwilson

      There’s a ton of work to be done and being doneWe are funding some of it in our 25ish portfolio companiesI believe there are big returns in figuring this all out

      1. Jon Michael Miles

        So today where are the services that tag or embed code that monetizes content where ever it may propagate? It would seem an opportunity for someone to develop a standard tagging that could be interleaved all content, so that every where – text, video, audio, games may propagate there would be a revenue stream to the author.This could be an industry standardized method, or a for profit model. Personally I like the for-profit idea and would be happy to pay a percentage to have someone track and help me monetize my text, audio and video content that displays somewhere someone is making ad revenue.I’ve coined a term for it – “DART – Digital Asset Revenue Management.”

        1. fredwilson

          Build itKiller idea

          1. Jon Michael Miles

            I was afraid you’d say that!

          2. Len

            Already done: http://www.ligertail.com = hybrid of Digg and AdWordsAs open and transparent as you can get, I think.

    2. kidmercury

      IMHO the solution is publishers becoming aggregators themselves, i.e. now that susan boyle has attention she could launch a service that aggregates music she likes, or could create her own “britain’s got talent” based on her tastes (i.e. she is the filter)the cost of creating aggregators is dropping to zero, so everybody can be an aggregator….now it’s about becoming the most trusted aggregator in your niche, in my opinion, and publishing great content is a great way to earn that trust

      1. fredwilson

        My post today is sort of my attempt to do that

  11. markslater

    foodista as well. What we need to do is to close the loop so that restaurants can participate meaningfully in this ecosystem. There is a huge amount of latent interest in food, eating, cooking and going out – if we can tie in a geolocating aspect, a one to one communications tool for the restaurants, and an open API for mini apps like – submit a recipe and have it featured on your favorite restaurant menu type stuff – then we have activity stream around meaningful context, and inherently social by nature. Enough of this ‘social networking’ – people will start gathering and acting around real interests and likes -and away from the web per se. (using tools that obviously connect to the web but are inherently mobile)We are in alpha in 2 restaurants in boston on a concept that attempts to bind all of these peices together. We should be in 30 by august.

    1. fredwilson


  12. Michael

    Did you know about e! Science News – its kind of Techmeme for science – http://esciencenews.com

    1. fredwilson

      No, I did notThanks!

  13. Jon

    I was looking for a program aggregate a few things that I like I found thishttp://www.mozenda.com looks like something that might do it.Anyone try any of these content aggregating softwares before?

  14. Steven Rosenbaum

    This is a great post, and the comments are spot on. Aggregation is powerful and important. And while technology can and will continue to do a better job of finding threads – humans (either crowds, or friends or professional organizers) are going to make the key difference. The one thing that often goes unsaid is that while those of us who work in media/web/digital stuff spend a ton of time searching and listening, for most folks the signal to noise ratio is already way out of wack – and that’s about to explode again when the iPhone gets video capture and upload sorted out (June?).The thing about finding a site that does a good job aggregating what you’re looking for – in the style and tone that fits your taste – it’s a real ‘WOW’ moment. Here’s a Readers Digest Published “Food” Agregation example: http://videos.tasteofhome.com and here’s their take on the “Health” space http://video.bestyouhealth.comThis is a space that’s just getting sorted – lots of innovation and exploration on the horizon.

  15. Kevin Prentiss

    This post is applicable to your #hackedu thread as well. Schools are publishers, aggregators and distributors of content.Accreditation, like journalist credentials, provides short term protection, but “Here Comes Everybody” with their own content.Publishing is commoditized at zero, leaving a huge opportunity to disrupt the distribution with aggregation and better filters.Tapeworms, indeed.

    1. C Seibel

      Great comment, and I would suggest that more than jsut schools we need to consider that school districts and schools, teachers and students are publishers, aggregators and distributors of content.As we begin to think about learning resource “management”, how far out of the box will we need to go in building the technical architectures to support “Here Comes Everybody”? School districts, and more broadly the government authorities (be they state/provincial, local or national), may not have the tools or access to the tools to make this happen.The decisions we will need to make today about learning resource management must be disruptive, and the publishing community will need to be ready to move as quickly as the schools will need to go.

  16. Logan

    So how again to publishers — a/k/a the creators — make enough money to cover creation expenses when someone else hoovers their content? At some point, there will be a chasm where paid media used to be. And people can say good riddance to old line MSM, but even new online-only content providers are getting ravaged by aggregators. With ad networks driving down CPMs, it’s getting much harder for all but the largest online destinations to turn a profit on content.

    1. fredwilson

      That argument doesn’t hold water to meAggregators send their readers to the contentThe content creators get to monetize it with advertisingThis model works well and thousands of internet content companies are doing well with itThe only ones who are not doing well with it are old school content companies that have bloated cost structures and mistaken ideas of what their content is worth

  17. Phanio

    To aggregate or not too aggregate? – that is a question that should be left up to the consumer. As the consumer’s habits and wants are what shuld drive the content on the internet.

    1. fredwilson

      Totally agree

  18. kevinmurphy

    on a side note- this post on aggregators proves there is so much room for the improvement of the internet experience. Fred’s reference to tapeworms resulted in this post displaying with a banner ad from google ads about removing parasites from humans… hummm…

    1. fredwilson

      Contextual targeting is not a great advertising techniqueI think profile/behavioral is much better

  19. Michael Lewkowitz

    The step beyond aggregation that I’m interested in is having the web present itself to me, based on who I am (what I’ve presented to it). Aggregators are an interim step. @tdavidson touched on it with a post on “personal API’s” that I’m hoping Taylor can post the link to. Public micro-messages (the most basic communicable element of the web) and in-line tags (the most effortless/basic form of explicit tagging/participation) are core elements in that. Twitters popularized the beginning of this (now the #1 source of web content I visit) and it’s opened up what I think is a fundamental shift in the use and form of the web.

    1. Michael Lewkowitz

      btw… here’s the link to Taylor Davidson’s post on Personal API’s http://www.unstructuredvent

      1. fredwilson


    2. fredwilson

      Is there an example of a post aggregator based on personal apis, micro-messages, and in-line tags?I want to experience thisIt sounds directionally correct to me

      1. Michael Lewkowitz

        not that i’ve found yet. we’re working on it and I’m sure there are (or soonwill be) others…

      2. jolan

        Isn’t that what InfoNgen could do or does for the financial sector?

        1. fredwilson

          Yes, but they are not a consumer facing service. They could enable that service easily

      3. William Mougayar

        Fred, I will email you on this.

        1. William Mougayar

          Fred, I’ve sent you an email with an activation link to your personalized dashboard super aggregator with semantic indexing, using the Eqentia platform.

  20. BmoreWire

    that’s a great site, now on my igoogle.In the same line of thinking, I really hate the subscription model that some newspapers are going in. For one, it’s not so much the money but the inconvenience of signing up for all of these different news sources in order to get news. So that’s my thought from the consumer side.The other of the coin is what you call “earned media”. Newspapers get a lot of their most compelling content from those people that are trying to “earn media” with PR or pitching news stories to them. So recently, the Baltimore Business Journal interviewed me and wrote an article about my company Socialroster, and other offshoots of Advertising.com (all internet companies). I was excited about this for several reasons. One that it promoted my company and had a great quote from my interview. But also because it promotes the Baltimore tech startup scene. They printed this article in their Friday print version and put it online but only allowed ‘print version or paid’ subscribers to read the article. So they literally reported on online-only companies and made it only available to ‘print-only readers.’ This is also the newspaper that broke the Advertising.com being acquired by AOL story which got them a ton of links and traffic and quoted them as the source which I’m sure drove their paper a lot of traffic and interest. The result of our article was that we got no jump in traffic, nobody saw the article and asked us about it, and we also couldn’t forward the article to any partners, investors, or anyone so in my opinion the article was a complete waste of time.If a journalist writes an article in the woods with nobody to read, are they even a journalist anymore?

    1. fredwilson

      That happened last night with Jessica’s story in the WSJ about Facebook’s changesIt was behind the pay wall and even though I have a paid account with WSJ, I didn’t want to bother with the login and two clicks it tookI just went to one of the facebook blogs and got the story that way

  21. Victor


  22. AnujMathur

    the value of aggregators can lie not only in directing us to relevant / popular / important news items, but some actually do a great job of structuring their aggregation. i think the realclear’s (RealClearSports / RealClearPolitics / RealClearMarkets) do a great job of this.

    1. fredwilson

      Yes, I left out RCP because I’m not spending a lot of time reading about politics right now but when I do, its my first stop

  23. David Buck

    As former (yes, I accept congratulations for my escape) web publisher for two newspapers, I agree 100%. And, not a small addendum, newspapers seem to forget how much of what they publish is aggregated content too!

  24. MikePLewis

    I love this post. Good one

  25. David Semeria

    Yes. And pretty much in-line with my own slightly Utopian views on the subject: http://lmframework.com/page

  26. alancommike

    Most, but not all, of the traditional media outlets in various niche markets (e.g food publlications) are now finally online. That’s providing a cross-over aggregation where the target audience straddles new world web services with old world print publication. It’s a little stretch on the typical thoughts of aggregation, but I think it fits.This straddling is easy with food/recipes because the user wants to go back to previously seen recipes. The publishers started doing this with just their properties, CondeNast’s Epicurious and Time’s myrecipes. Of course the next step is to aggregate across publishers, which is what ProjectFoodie is doing now.I haven’t thought about other niches, but I’d expect that there’s similar cross-over and may already be players there. As this matures, I expect the various niches will consolidate on common platforms. …alan

  27. Carl Rahn Griffith

    As an avid consumer of news I don’t care about the source of a story but rather the quality of the article (ie, the research/the writer/s) – sure, I initially associate some gravitas to anything i see that originates from (eg) wsj.com, economist.com, techcrunch.com, guardian.co.uk, etc, but my ultimate judgement as to the merits of a story are in its content, not where it was originally hosted/published from – the automated discovery and presentation of stories/news relevant to me is what is key – and where the real value lies … this is where aggregation has so much potential …

  28. William Mougayar

    Old media is furious because most of them haven’t been able to successfuly migrate to social media.The modus operandi of (most of) their writers doesn’t allow them to be fully participative in the conversations of their readers because the pressures to move on to the next story.

    1. fredwilson

      that is so right, but if you don’t participate in the discussion, you’ve really missed the whole point of social media

  29. ravisohal

    Aggregation is breaking down content – in a good way for users/consumers. When we aggregate content we end up creating useful information verticals – so we get local restaurant aggregators and menu aggregators (eg foodiebytes.com). It creates a lot of contact points for the publisher. It also changes the fundamentals on how this is monetized – which I think is still a bit up in the air. Probably why the big media publishers are queasy.

  30. deeba

    This is a great post. I love Tastespotting. It’s a great aggregator…very droolicious too!

  31. fredwilson

    Sure, google is the tapeworm that the WSJ is most upset with