Why Social Beats Search
That's a controversial post headline and I don't mean that social will always beat search, but there's a rising chorus out there about "content farms" and search optimized content creation that is worth touching on.
Arrington started it when he posted about "the end of hand crafted content". Richard MacManus penned a similar post the same day called "Content Farms: Why Media, Blogs, and Google should be worried". And over the weekend, Paul Kedrosky addressed the issue of search spam in his quest to find the perfect dishwasher.
When a web service like Google controls a huge amount of web traffic (>50% for many sites), it's going to get spammed up. Google has thousands of employees working to combat that spam. And it is doing the best it can. But it is hard to beat spammers. The best you can do most of the time is tread water.
What's worse, and what Mike and Richard are talking about, is the act of search engine driven content creation. These so called "content farms" look for the top search terms and create content especially for them. Arrington compared this content to fast food; cheap, easy, and ultimately unhealthy.
I left this comment at the end of a very long comment thread on Arrington's post:
social tools will allow us to decide what is crap and what is not. our social graphs will help us. search engines won’t. it’s a lot harder to spam yourself into a social graph.
The Internet is a massive content creation machine. We are witnessing an explosion in the amount of content getting created every day. Most of it is garbage. But some of it is not. Machines can help us find what is good. But with the help of machines, our friends and trusted sources can and will do that even better.
Spam touches everything. Why should search results be magically immune?
I agree 1,000 % and this is exactly the theory Whitney and I are building our wow.ly “Tools for Social Data” projects on … our latest in beta is http://grou.pe which let’s you pick up to five Twitter users that interest you, and it will report back on who those people as a group are all following that you aren’t (ie. if five journalists are all following someone, there’s a good chance that’s a really relevant person for journalists or people interested in what journalists are being exposed to, to follow).Anyway – love this train of thought and the open discussion around these approaches! Thanks.
Kevin and Whitney’s work with Grou.pe is pretty cool. Definitely recommend checking it out for anyone interested…
Nice leveraging of existing data by connecting professional and information channels!Great stuff.
Man, you beat me to the comments. ;-)The other factor that’s worth mentioning is that the combination of “machines, our friends, and trusted sources” that you [Fred] mention hits an issue that I think is key.”Curation” is a buzzword right now, and I love the actively curated streams I get via Tumblr, Twitter, and related, but I’m also fascinated by the potential of active/passive creation–letting machines help.Both conversationlists and grou.pe — which I will continue to call Hivemind, by the way 😉 — are based on the idea that what people are already doing constitutes a sort of curation. Conversationlists takes who you’re paying direct attention to as curation, Hivemind takes the intersection of people’s decisions about who to follow on Twitter as curation.In both cases (as well as a few others we’ve got queued up) adding a machine that pays attention to the flow of the data that people are creating lets us find and expose the “curation” that people are doing without even thinking about it. It isn’t a replacement for manual curation, but rather a supplement. It’s good fun with some great potential.
In other words its like adding a smarter filter vs. getting the full stream
Yep, with the caveat that it’s most interesting and powerful when we’re talking about a filter that’s applied across multiple sources. Finding intersections and congruence, not just removing noise from a stream.
Putting humans and machines together is always better. The trick is to do that in a way that the human part scales
Absolutely — and it seems like that’s an area that’s had some fascinating developments recently. Between tools/services that minimize what people have to do in order to “contribute” and those that are finding ways to convert what people are already doing into contributions, there’s some great stuff happening now.
Great idea. Reminds me of some of the older expert-filtering systems used in general search – like the Hilltop algorithm (http://bit.ly/4sjO1s). Except in this case you’re filtering by actual experts instead just expert documents.
This only works if you actively maintain your social graph. I get followed (and subsequently get unfollowed when I don’t reciprocate) dozens of new spam twitter followers a day. If I was automatically following everyone who follows me (which is quite common) my social graph would become useless as a filtering tool.I think the public listing of how many followers one has is the single most destructive thing for social media for discovery because it encourages people to follow without thinking just to increase their public social footprint.
I agree with you that people follow too many people. But I think pruning and optimizing our social graphs is going to become an important skill
Getting over the “popularity contest” of the internet makes it a lot better.Success is quality over quantity when it comes to Twitter, LinkedIn, Facebook etc…
Great comment …couldn’t agree more
I agree as well. I don’t think it’s going to be long before you’ll be able to start grouping the people you follow (I think tweetdeck already does it a bit as well)…so you can have your social network segmented into your interest like ‘car buffs’, ‘programmers’, etc.I *think* that was sort of the driving force of Twitter lists…and I think it will only evolve into a much more useful set of features (and way of thinking) from here.
Twitter lists will become more useful. I first think we need to discover what kind of uses there are for them. I was having a discussion with an art friend of mine, and he mentioned that Twitter is a medium, by extension, the lists are as well.How many uses for the lists can you come up with?
I’d like to tag people more than group themDelicious and gmail are the formative user experiences for me around categorization systems that scale
What is grouping other than reverse tagging? I mean I can either tag a bunch of individuals as ‘VC’ or I can put a handful of people together and tag that collection as a ‘VC group’…but otherwise I think we agree.To me, the coolness of something like http://grou.pe isn’t the group, it’s the passive tagging that occurs via the group (ie. instead of having to tag 5 people as VC, and then see an intersection based on keyword…I can just group 5 people and see the intersection — it’s less actual work and allows my brain to recognize the slight variations in interests that a ‘group’ has vs. just a strict key word or two)It might just be me, but I think the brain more naturally groups than it tags…even though we probably *think* we are tagging, we are often actually grouping things in with something else it’s “like” or something else we understand…
In theory you could use the lists like that…What would a streaming tag system look like to you? What is the difference between a list and a tag beyond semantics?
The Wisdom of the Language ( http://sn.im/wisdom-link ) both tags and groups. See also the examples I gave (food.com + birds.com) in my comment below.:) nmw
I think this is an area where machines can help. By extracting better meta-data from articles and blog posts and using it to filter our connections with different people, we’ll be able to follow more people and still get better information.On Twitter, I follow some people for business, some for music information, some for pure entertainment. Yet no matter what my purpose for following them, I receive all of their updates, much of which constitutes ‘spam’ for me. While I appreciate that the business contacts I follow have every right to tweet about what they had for dinner or how well their kids did on a test, giving me a way to filter that information out will be crucial if social networks are to become precision instruments instead of blunt tools.
what you are saying here implies a complicated question about what is what for some people.One of the most brilliant ideas I ever heard of is Spoetry.It takes machine created content and turns it into human content: Spam as art in the most random, John Cage like chance way.I find it inspiring. Perception of what something is is incredibly important in media. Everything, including original and non-original content, is blunt until a person gets involved in its curation as a medium of partial creation. What that curation should look like and be like is complex and nuanced. The idea of the meta-creation, where we think from something previous, is in no way bad nor good: It is just a state of investigation about content and thought. All we have is information, it is a matter of how we are choosing to process it under different circumstances.Your perception of what matters is how to go about the curation. Maybe it should be by machine, maybe it should be through human thought. We just need to perceive the computer on the other end “well” for our needs, whether it is art, or to find a good washer.
Perhaps Plato can be a useful reference here: His theory of perception goes something like this- Everything is perfect. Even if I see something, or sense something that to me is imperfect, it is simply my vision that is imperfect, and the object which I see is intrinsically always perfect.
Phaedrus or Republic? Plato is a weak spot of mine ;). I actually got a fabulous professor for when I the read the Phaedrus…one of the best professors I ever had. I should sit down and read Ion one dayI would say that is NeoPlatonist than Platonist…just because of that one professor. One of the more interesting reads of the Poetics we did was of the nature of recitation: The end of the Phadreus has Socrates rejecting writing because it decreases parts of memory. Once it is set down, it is harder to engage in. Form in this conception in just as much a function of the mind before it is a function of the object.One note: My first education in a serious read of Platonic dialogues were of the Strauss/Bloom school of how to read Platonic dialogues. Not the same professor(s) -but for the sake of disclosure.
I find it hard to follow more than few hundred people per list. I don’t auto-follow but rather look if my list followers and especially in their own lists offer information that I missed in the first place.Whilst I can’t see Search being replaced, Social Discovery offers me so many other ways to discover great information if the lists are updated/pruned/optimized all the time; it’s never perfect but getting better over time.I enjoy creating lists of places and topics that interest me and friends and to give them a ‘home’ with their own domain name as it makes it easier for me to go to a specific destination/topic of interest that I can build upon.I find myself more and more to do search and then check Twitter Lists and compare them and there are unique findings on both sides all the time. I can’t look through books on Twitter but I also can’t see what the entire team of NASA astronauts are up to on Google. Now I can do both and loving it.
Fred, I predict that in about a years time, you will be overwhelmed with so much commentary that you will no longer be able to read most of it.You will have to put up a wall of some sort for your own sanity or drop the blog altogether. You will probably put up a pay wall to keep only the most interested- you will donate $100 % of that fee to some charity.just a thought
An ecclectic bunch of people will yield an ecclectic bunch of content- which is good for discovery but not for going deep within a given field of interest unless you follow a homogeneous list for that topic. So people-tagging, filtering and list management will be needed.
Can we use algorithms to understand who is important and who is not on a relative basis. From there you can determine whose content to see and whose to suppress or at least limit until you have read all the stuff from people you care about.Obviously “who is important” may vary from context to context, when it comes to the social graph, you can get pretty far in determining relative priority via communications history (e.g. email and meetings) and “connectedness” (LinkedIn, Facebook, Twitter…).From the “important people” you can discover and deliver one specific content feed. This approach will not solve the “diswasher” question, but it will help in discovering valuable content as the prioritized social graph is a very good first pass filter.Would love to know what people think in one of many implementations at http://www.gist.com
I see this as a self correcting issue. You really only have X authentic followers, and Y people that you really follow and interact with. Those that have 100k people in their input channel have flooded it with noise. And it’s likely their followers are ignoring them for the same reason.Shrink your input stream to a tiny footprint, and list the rest of the folks you have interest in. That setup is working marvelously for me.
Well put. On my public Twitter account, I follow back anyone who follows me. I know there are a ton of spammers, but I’d rather reciprocate. I just dive in and find interesting stuff to add to my filters, and as a result, I probably only have about 200 people on that input stream.Great comment.
Thanks Aaron, I used to reciprocate following back but at some point I realized I was giving too small a slice of my attention to everyone. That lead to the “great culling” of ’09. Sad as I was to unfollowing many, the option of lists fills the data void nicely.
It probably won’t be just who you follow, but also how often you interact, quality of those interactions, reputation of profiles over time, etc – applied iteratively across the entire social graph. Plenty of cues available to separate the signal from the spam.
What you say is right Fred, but unfortunately my SN can’t answer all of my questions, and even if it could, if I asked too often I’d get unfriended pretty fast.
I don’t see this as an active query of your social graph (like a google search)I see it as passive following of link sharing and other social gestures
I like passive social discovery very much, but you can’t compare it to intent-driven search. For me, they represent two very different use cases.
I think there’s another option too. What about intent-driven search with the results weighted by social graph information? Best of both worlds?
I think you’ve got a great point, but I also think the key isn’t just ‘your’ social network…to me it’s the concept of social networks as a whole…often times I want to ask my own friends something, but other times I want to ask a panel of experts something…in those cases, I think being able to access a social network of experts is where the power comes in (ie. what if you could access a group of 5-10 computer science teachers and listen in on what they are talking about as a group?).I also think it’s important that the ‘asking’ is done more passively…and that’s done by being able to access more than just real time, but going into the archives of a given social network…being able to filter, merge, flip, shove, and move bits of conversations around to help you find new perspectives and ideas on your own (ie. without just pinging an expert with “hey what do you think about X”)…
Yes, Marissa Mayer touched on an similar idea during Le Web. She was talking about ranking search results based on possible results from your social graph. Your idea of including other but nonetheless ‘respectable’ graphs is an interesting extension to this idea.
What is a social Graph? Most of it is loose ties. Most of us have overlapping needs? Even the idea of a Graph is complicated to someone like me. A grid in the artistic field reminds me too much of the Bauhaus, and their ideas regularity of thought and form. I want certain people, some of whom I don’t even really know to inform certain elements of my thought, because it is an additive experience. If I interact only occasionally via the follow with someone on twitter about social media, religion, and racisim: that is positive for me. I understand something more than I did before. My relationships with people ebb and flow like water into a river. Grids can be very unhelpful to map that form. All sorts of good information may be occasional- the question is how to we filter out the dirt and chaff from the water to make it pure.
The archive retrieval is an issue with relying on Twitter search (in a tool I’m helping develop). We opted to add Google social search to find real time information outside of Twitter in addition to older shared topical data.Good reminder on limitations of decaying real time data.
I think that this is an interesting corollary … this idea of panels of experts. Twitter lists begins to address some of this need, because not only can I build my own lists of trusted resources in a certain area; I can also use lists that people I trust have already created.The “list” phenomenon is still in its infancy (or at least my understanding of it is), but with tools like TweepML and Tweetdeck incorporating Twitter List functionality into their tools, we’re going to start seeing more really targeted and valuable networks of people.
My favorite list right now is conversationalist because it is dynamicStatic lists remind me of folders in outllook – a 20 year old idea that’s come and goneI want dynamic lists that change all the time. And I want tags
Sorry just catching up on some of the older threads now – awesome that you like conversationlist and I agree with you on the need for dynamic list features (in fact, we just implemented dynamic lists from Hivemind results today as well at http://grou.pe — the second little tool in the wow.ly umbrella after conversationlist).BTW – you probably don’t know about it, but we also have some stats and things we are starting to share with people on conversationlist … for example you can see what Hivemind’s you’ve run or been included in, and you can see what conversationlists you’ve appeared on and who’s appeared on yours over time…it’s all available at http://wow.ly/fredwilson (we gotta work on the presentation of all theses stats and things still)…We also already have some really interesting stats on who’s on conversationlists as a whole (all time and daily) who’s got the most changing lists over time, etc. etc.Some really interesting things you can do with all the extra meta data that comes from simple little services like conversationlist and hivemind (and we’re hot on the path to releasing our third little tool in the next week or two as well)…
Awesome. I’ll check all of this out
searching the archives is way more scaleable than looking for an answer inreal time
Archives are indeed good, Fred – they stand the test of time.A lot (way too many) of ‘real-time’ people on the ‘net spend most of their time pontificating/speculating/’blobbing’* (*Copyright, my wife, Helen), etc, about ‘the future’ knowing full well that’s a VERY easy way to sound like a visionary and get a few ‘cool’ sound-bites re-quoted, with no accountability or empirical referencing required.I love the optimistic pragmatism here. I suspect many of us do.
“the key isn’t just ‘your’ social network…to me it’s the concept of social networks as a whole…often times I want to ask my own friends something, but other times I want to ask a panel of experts something”Totally agree Kevin, I think it’s about extracting information from your social graph and using it to build a generalized reputation system that anyone can leverage to find trusted answers.It’s more about find information from people that are trusted by someone, not just people you yourself trust.
I’m a pretty determined information seeking agent. If I believe there’s a social network out there that collectively has the knowledge I seek, you can bet your bottom I’ll beg, borrow and work my way in.I do rub some folks the wrong way, but there’s not enough sands in my hourglass to stop moving.
Awesome piece. What’s interesting is that social networks are for the most part decentralized. While the service may be a central service the searching is being done by a decentralized network of people which you yourself curate. Contrast this with Google where the content may be decentralized but the search centralized at Google. So social could be a reaction against this and be seen as the re-decentralization of the web.When you know what you’re looking for search will always be king. When you don’t know what you’re looking for but just looking, social is where it is at.
Great comment too
I think it depends on the question…who better to ask than your own friends when the question is “I’m bored, anyone got something fun for me to learn or read?”…or even “what’s the best dryer out there?” (if I’ve got friends who’ve bought one or had to read up on them recently, I much prefer to see their sources than a random google pointer to sites about dryers).In fact, I find that more and more my google searchers are being dumped into two groups these days: one for high-level, quick overview, “let me see if there’s anything interesting about X online” sort of searches…and one for very specific technical answers (because I’m a programmer) “Perl Hash of Arrays sorting keys” sort of deal…Almost everything else I either rely on my social network to bring to my attention passively OR I specifically search through a social network (not always my own) asking questions and looking for answers…Indirectly, I think this has been around a long time and is why things like Amazon reviews are so powerful and helpful…
Definitely. Every question is related to a problem. In Google’s case the question is “What are you looking for?” The Google algorithm is designed to solve that problem and the users with the most informed answers to that question will find their problem best solved.In the case of Twitter, everyone is answering a broader question, “What’s Happening” or more generally “Who, what, when where why how?”. It’s the user goal of discovering people who are likely to have informed answers to problems they have or may have at some period of time. On Twitter, the users are the search algorithm.
Agree. Wrote this post last night on this subject. I love the idea of ‘McDonalds’ content. Not all calories are ‘good calories’. Not all content is ‘good content’. The problem for machines is that they have yet to pass the Turing Test so we can hardly expect them to distinguish between good and empty content calories. http://thebln.com/2009/12/m…
Thanks for the linkI’ll go read your post
Yeah, I think many winners in this game will be future people and businesses that gain the respect from a large population of people of doing a good job of seperating the crap from the good journalism, including investigatinve journalism which media farms can’t seem to do well. Think the ABCs, CBS NBC of the past, or trusted anchors such as tom brokaw in news or Opra in interpersonal sbjects. These people and firms will pop up in the future and be very expensive.Back in the cold war behind the iron curtain, there was alot of propaganda radio (the cold war equivalent of media farm content), but guess what? The majority of people behind the Iron curtain knew it was crap and -anyone who wanted serious and acurate news knew to tuned into the BBC and VOice of America broadcasts because of a superior reputation for reliablility and accuracy.
Bingo, Bullseye.Social search is how I’ve been consuming big steaming bowls of information over the past year. And the adoption of social search has only picked up. Big automated super search engines have made moves to integrate social search into their baseline because they realize how potent it can be.Happy users, return to the map that found them the buried treasure. The next evolution is cultivating your own mini search agent (not engine). Google, Microsoft, Facebook, & Twitter all know this.There’s a wonderful space where social overlaps with automated tools. I call it intelligent virtual assistants, and want nothing more than to see them blossom into the valued tool of 2010 and beyond. Tools are capable of remembering my friends, my inluencers, my competitors, my passions and my taboo subjects. Blending complex, interactive and dynamic social channels is akin to artistic information sculpting.
I think you’re absolutely right. The fight against this kind of spam is very difficult, and in the end it’s going to be a race between Google and the content farms to come up with new screening systems / ways to get around them.What if Google used social graphs to improve the search results?
Now there’s an idea that’s worth working on both inside of and outside of google
I’ve always thought it would have been much more interesting for Yahoo! to sell their search business to Facebook rather than to MSFT.
If only more ambitious value targeting, data mining junkies were working on this “BIG” opportunity.I know of a solid 5-6 that will likely be many times that by the end of next year. The greatest aspect of inevitable technology is that you can feel it.
It is so difficult to do this- you said you wanted an anthropologist. Ask an anthropologist or a sociologist about how to define community, especially when you talk about the margins. Especially now. Especially as media and medium changes how community interacts.Who counts as in and out and can force change onto a community in practical terms has yet to really blossom. You are starting to see real life examples, but it is really difficult to think of massive compounded social changes and how they worked out: Say marriages. We don’t know yet.
It does, check Google social search, True or False Dichotomy. The beautiful part about this challenge is that it’s open, and the solution will have as many “optimal” answers as there are communities.
Yes indeed, they do. Thanks for the link.I was thinking more about them incorporating some “social” aspect to improve the search results in google.com, not in a special section. As you say it’s a huge challenge, since there will be different “optimal” answers for different communities, but it could at least be another weapon in the arsenal against spammers.
Joaquin – I’ve been thinking the same thoughts for the last year or so. Just presented a talk at SES Chicago on ‘Social Search and Distributed Reputation Systems’ – is this similar to what you’re thinking? http://bit.ly/4YhXq4
Yes, I was thinking along those lines. It would be difficult to implement the ID verification you’re talking about in the presentation in a universal way. The best candidate that comes to mind is Facebook Connect, but I don’t see Google embracing it over OpenID.
Yep – both Google Friend Connect and Facebook Connect got a callout on my presentation as the most likely solutions now that they’ve become well distributed. I’ll still root for the OpenID/open-platform solution, but they just don’t have the resources to compete at this point.
Agree – one of our key milestones in 2010 ripple100 – the role of social/content farms in marketing/ads. To tell us with reasonable consistency and relevance: where do I go if I’m interested in (interest area) in (location) for (demographic)? I don’t see how machine search can do that. But we’ve drawn up a way for social/content farms, mayored by opt-in influencers, can do that.
Fred, I largely agree. However, social needs to get categorized. I am experiencing the context problem right now. I speak to a general, professional-biased audience in twitter, a pure professional one on linkedin, and a purely social one on facebook. friendfeed ties this all together. i let twitter update my facebook status, which makes some of my friends very confused, since they don’t understand twitter syntax. this is just an example of the problems.having said that, i fully expect these problems to be sorted out. unlike industrial revolution era systems (like telephones, or even mail), top down taxonomies don’t work well with the social graph. the wisdom of crowds will solve the primary problems. however, for that to happen, the platforms need to be flexible to allow that type of self organization. twitter’s been good at that. facebook, less so. let’s see if the push towards openness contiunues.
Right on, there’s this user driven emergence of acceptable information standards. It changes rapidly though, as protocols adapt to more rapid data exhange and stronger/deeper social meta data.We’re teaching the web how to treat our social graph, our shared content, our greatest interests and enthusiasm. The collective web is learning what we value and giving us a network to optimize it.You certainly get it!
Mark”The collective web is learning what we value and giving us a network to optimize it.” I like this alot but right now it seems more like an idea to work towards. Info gathering is still hard work as the networks aren’t sorting efficiently for me.It’s a balance between searching objectively and vetting with communities. Or pulling info from a conversation and vetting it with the datapoints somewhere else. Back and forth with myself as the network in between.Crowd sourced info portals on topics offered some hope on this but building dynamics across multiple unconnected communities was the big fault here. Intelligent connections have to hold the answer.
And maybe Mark, blog communities like this one are the ‘intelligent connectors’ that sort our conjecture and fact into digestible trusted bits of socialized fact.
The evidence of our conjecture is when it’s built by open source teams, founders, and the like.Part of my drive for building something of lasting social value is attracting the true engines of tech startups, genuine engineers.
I’m a believer Mark and doing my small part to drive this to reality every day.
What is an objective search?
My bad on choice of words. I meant ‘non socialized’ Google-esque search.
I agree with you about the problems and that we’ll solve them
Bravo. …the cycle here of discovery, I think, as we progress through the paradigm shift…soon quality will be based on nourishment and satiation again. The hand made content that is easily found and invites you into it’s warm fireplaced virtual living room to inform and entertain you and tell stories that you can interface on a human soul level… will have a place. Fred your blog is a very good example. Obviously quality can give way to massive crappy production…but there will always be a hunger for credible quality. People would buy the big turkey dinner if they passed their Mother’s kitchen table on the way to the counter in BK or McD’s..especially if they are cheaper or cost the same. The question that these large historic corporations have ..and the new media people who have come to fame for just being at the “first to the plate” not based on quality of their talent or knowledge.. can’t understand…is how you do that. Handmade quality in content is stronger, tastier, more satisfying a fix and therefore more apt if accessible to be just fine. inmho the internet and cheap new production technologies…have set up one of the only times that fast food isn’t cheaper and easier to get than that Thanksgiving meal.
The hand crafted movement is gaining ground for a reason
For an example from the burger space (though I don’t think anyone in the burger industry would call it a “space”), see Five Guys (hand-formed burgers, nothing frozen, high quality control), which has been on a tear since it started franchising. Eating an occasional burger at Five Guys is one Obama policy I support.
100% True. Fred, this very blog is a great proof to what you are saying. This post, and the comments provide invaluable insight that can not be found even in hours of (futile) Google search.I think what Arrington really saying is that people consume crap. They always did, always will. This is why we have the Sun and Jerry Springer (i daresay that clicking a virtual cat and select “pet” from a menu is crap too). This is similar to the old rating dilema, now on the web.
Does the difference lie in the fact that those that don’t consume crap now have a plethora of “non-crap” to sift through? You can’t paint everyone with the same brush. Not all people eat at Burger King, a lot are health conscious (and that trend is growing)!
Sometimes a 2 minutes short demands’ media video is enough. Or you just want to take a look at the headlines, no matter the newspaper, while eating at `Burger King’. That’s OK, as long as you remember to keep enough quality consumption in the mix.What I was trying to say is that this phenomenon is not new, and thus the prediction that the high rating – low quality will be the ONLY type popular content is far fetched, although these cheap content will probably be very popular.
the end of the information is being heralded here …algorithms reduce information to data …next up, the wisdom age … social media is not that, yet
“Wisdom Age” – what a great term.It’s definitely becoming clear that the human element is the best mechanism for sifting through the noise.
But you have to pick the right super human filters. Each of us has unique tastes within our fields and topics of choice. Very few of the folks I socialize with offline think about this space at all. So here I come, to listen, learn, and share my findings.
There’s some interesting parallels here with early expert re-ranking systems like the Hilltop (http://bit.ly/4sjO1s) and HITS (http://bit.ly/3sRFFh ) algorithms. Wonder if similar thinking could be applied to automated filtering via actual human experts?
Groovy, we can begin to imagine analogs to these algorithms in real time social shares using likes/retweets as a measure of approval.I want there to be information attention gravity to the posts that would benefit me the most with actionable, or fascinating info. That’s why I’m working on real timesemantic search, it’s a lead in to intelligent learning systems.
likesretweetscommentsclick-throughswall postsDM’scall records (Google Android…)There’s so much useful data out there once we step beyond the limitations of the link economy.
“It’s definitely becoming clear that the human element is the best mechanism for sifting through the noise.”In a way it always has been – PageRank was the first social search algorithm, it used social annotation in the form of links to filter the web. Before Google it was all META tag spam and auto-generated content pages. The fact that we’re seeing spam issues again suggests we need some new and improved social filters to keep the web manageable and usable.
Is the wisdom age when Skynet will achieve self-awareness? 😉
Ha Greg, it’s when we stop wasting bits chatting about it, and go build it!
As soon as social becomes just as effective as search at capturing INTENT, then you’ll see some serious disruption occur. I feel as if social is a passive method of content discovery, whereas with search, the customer displays his/her intent to find a particular piece of content (product, article, etc.) and therefore, the ads displayed around it perform well.While I haven’t used it myself I’m wondering if social search service, Aardvark (vark.com), is a solid step in that direction.
In regular search you capture intent whenever a user starts a search and display ads or other relevant content (including search results) based on the query.In social media applications you capture intent whenever a user *shares or somehow enhances content* (retweet, reshare, favorite, bookmark, and so on). Who’s doing anything relevant with that information right now?
Social sharing adds authority to the shared content & source, but I can’t see what intent it conveys.
When you share a link or comment on a post like this one you’re showing that you’re interested, right? Someone can write an application that will react accordingly and suggest you some related product (advertisement).Also, if you go into natural language processing there’s so much more to capture. Expressions such as “Looking for…”, “Love this product” and “Having lunch at…” convey intent. These expressions give opportunity to direct quasi-real-time advertisement where some application can almost instantly reply to the user with some related information.
Hmm advertising. I suppose I should be thankful for all the free services it supports, but still….
Not only advertising but also the ability to suggest other links you might find relevant, hence helping you through your daily reading.
Bruno – This is a really interesting idea and one that I’ve been thinking about lately. I would make the argument that there are really two types of sharing that convey different levels of intent. The first is value-added sharing, where people leave comments, write a post, or otherwise enhance the original content by adding context, an opinion, or something else that someone might like.The second is a simple share, such as a reblog or retweet, where the content is simply being moved along and disseminated to a wider audience.I think that Twitter will begin to capture some of the value inherent in the latter form by providing some sort of monetization strategy around trending terms, content, etc. But the former category of value-added sharing occurs on niche content sites and personal blogs. Huge amounts of value are being created here, and I’m wondering who (or what) might arise that will be able to effectively capture that value.I took a stab at this last week in this post: http://blog.pinyadda.com/20…
You can if there’s a shoutout in the share
Its a step but an awkward one (at least right now)
This step is awkward, but Aardvark’s next step will be powerful. The answers and understanding their building up is just waiting to be made public. I’ve gotten excellent answers on there, but nobody except me can use that value that was created. I think that’s why Google is making a bid for them.
Advertising is spam by definition, so naturally it works well when the info is pushed.
Social search applications like Aardvark are interesting but my issue is there’s too much of a fall off in terms of content comprehensiveness in using apps like this. I find Aardvark is great for getting answers to esoteric questions where you need an expert, but not giving you a subset of curated information to make a decision. There needs to be applications that shift the curve as opposed to force you in such a drastic trade-off scenario. Here is a blog post I wrote on this topic: http://blog.funnelscope.com…
Completely agree Fred. This is why social networks such as twitter and facebook are extremely important to Publishers. They have to figure out how to use these social networks effectively as they become an increasingly important source of traffic. Because those publishers that don’t become a valuable source of news and information within these networks will find it harder and harder to break in over time. Publishers have to think about what content works well with each segment of these social networks, and provide information their audience finds valuable consistently. And when that audience comes back to their site, they have to think about how to maintain some connection to that social space.There is a broader concern that develops as people self select into social/news groups that could serve as an echo chamber for thier ideas. But that’s a different conversation.
As the community changes so does their message redefine the boundaries of the information echo chamber.
Possibly. But if most people have a fairly steady state of friends and people that they follow, then that would suggest that the echo chamber will stay fairly steady as well. It depends on how we define community, and then how the players within that community behave. To keep a stream of fresh and new ideas flowing within a community, it suggests either bringing in new sources of news and information on a regular basis, or new members into that community.I know I actively curate my twitter lists, which keeps idea flow fluid. But I’m not so sure that most people do. And with facebook, many people keep their friends to a selection of people they truly consider ‘friends’.This is broader than just social vs. search. As news and information continues to find niches where they can distribute specific views (think Fox or MSNBC vs. ABC/NBC/CBS from 20 years ago), people will get news and information that reinforces their ideas. And historically, people have tended to gravitate to news sources that support their world view…which is getting easier and easier to do.
Dynamic social graphs are the next big thing in social. Figuring out how to make it easy for me to change my graph as I need to isn’t going to be easy but the payoff is huge
Now that is something to keep an eye out for. As you said, it certainly doesn’t seem easy, but tools to help manage our social graphs is indeed a key missing piece.
While I’m in agreement with you re: spam, I will say I’m not in love with people going straight to their social graph for an answer they could’ve easily found themselves.Luckily, there’s LetMeGoogleThatForYou.com – which makes answering those questions a lil’ more fun.
I have been the fortunate recipient of Reece’s LetmeGoogleThat message. Mobile’s aren’t quite there as far as optimal browsing experiences go- I appreciate the help even though it’s a commentary on my terrible intellectual sloth 🙂
Ha… True re: mobiles. In that instance, social graph can beat search.Still, LetMeGoogleThatForYou is always fun…
Great post, Fred. When you think about paradigms like the trust-based economy, you can see this sort of thinking fit in very nicely. I’m sure I’m not the only person who places much more value on something when a person I trust or respect has recommended it, and that’s an implicit level of trust I don’t (some would say can’t, though I’m not willing to go that far) have with a purely mechanized search engine.It’s fascinating how the human element consistently returns to areas where it was thought to be dead and finished.
a recommendation doesn’t even have to be explicitif i see a friend who has good taste in restaurants checkin to a placemultiple times on foursquare, then i’m gonna go there too
I can’t wait for Foursquare to make historical data more accessible.Seems like the guys are (quite correctly) working through privacy implications before going too far in this direction, but I would love to build have a FS app that shows me nearby places that my friends tend to frequent.When I’m in the wilds of midtown, being able to quickly see if there’s anyplace that Mike White and Andy Weissman go for coffee would be a jewel beyond price.
You’re absolutely right, Fred – I think that’s a big part of the true genius of Foursquare.
I had the same experience recently. Looking for a restaurant in an unfamiliar city, and I was able to pick a great one through the implied recommendation of seeing several people I knew through the social graph (but had never met!) had been there.I think this type of implied recommendation will eventually be very powerful for all kinds of goods and services. But the key to making this work across a wide range of goods and services is an automated mechanism to capture your preferences. Like Foursquare does for places.If you were allowed to opt in (so that it was voluntary) I would love to see all my book purchases from Amazon get automatically tagged and added to my social media accounts. If others in my graph did that, I would soon have a large inventory of recommended books to pick from when I next wanted to buy something to read. All with no conscious effort from my friends or contacts other than opting in.This could work for all kinds of things – goods and services, vacation destinations, restaurants, you name it.All it needs is an automated, or series of automated tagging services..
Comments on possible statistical limits affecting the quality of information derived from BOTH Small sized social groups (for example, a group of 10 people) and larger sized social groups (ex. a group of 100+)Assumption: One wants informed, valuable (to you) opinion and advice from the group.Problems of a Small Group of 9 people: A smaller group of your chosen peers is more likely to be in tune with your tastes, but also is a very narrow opinion leads to a lack of diverse opinions. Kinda of like inbreeding. Statistically, if you asked 9 people who they will vote for and try to use that to predict an election result for a larger population, your margin of error is 1/Square root of N where N is the sample size (in this case 9) result: 33% chance that that the survey result is wrong.Large groups Ex 100: Information derived from a larger network tends more accurately represent a diversity of opinion, and predict a larger group but you lose the expert opinion and results tend to be watered down and so average as to be devoid of informed opinion. The margin of error of the opinion of a group of 100 is 10%- more accurate than a small group but the result is dumbed down to an average.Where’s the balance? Who knows. But A similiar debate exist in the judicial world: If you are on trial for murder, and you are innocent, do you want a jury of expertly trained jurists (as in Britian) to jude you, or do you want a jury of your peers chosen at random from a large population to judge you?
I love last paragraph!
Social tools offer a push rather than pull signal vs noise filtering system and thus whilst helpful for pre-screening links I come across in my travels do not (as yet) offer me any assistance whilst performing acutely targeted and idiosyncratic searches which we each inevitably do on a daily basis.We easily decipher between good honest content and spam farm/affiliate scheme created content and act on the information accordingly. I don’t think it will be long before googles algorithms and mainstream Internet users can do the same. Hopefully alleviating if not eradicating current problems.( disqus on the iPhone is hard work )
This piggybacks off of Sean Parker’s talk this year at the Web 2.0 Summit “High Order Bit: The Rise of the Network Company” (http://bit.ly/2eSNO6). Agreed.
I did not see thatThanks for the link
Too bad all my friends are idiots.
Too bad I don’t believe you andy
Leads me to another thought: There are certain friends that you love to death, but experience tells you will always benefit by prefering the opposite of their advice. A $3.00 PRIZE ANNOUNCE FOR THE FIRST PERSON TO CREATE AN ALGORITHM TO ACCOUNT FOR THIS! Plus, I will through in a mccaarthur genius certificate bought off a trusted source on ebay!
A $3.00 PRIZE ANNOUNCE FOR THE FIRST PERSON TO CREATE AN ALGORITHM TO ACCOUNT FOR THIS!There’s an old investing strategy based on a similar principal: the odd lot theory.
Thank you as I was not aware of this-‘course its not quite the solution to my challenge but you do win the $3- Next time I see you at a pub you can count on a can of PBR from me. word!
Thanks, but instead of a can of PBR, maybe you can buy me $3 worth of something good on tap.
Absolutely! Anything on the menu! But please note, and confirm by your twitter community, PBR is the new Hipster beer- kinda retro thing i think !
Completely agree. I still trust friends and others in my social graph for recommendations far and above any system generated or crowd-sourced recommendation (generally). We are social creatures
I agree with the trend and it raises a few questions:1. What will this mean for the long-term value of Google and its organic search business, which supports its advertising side?2. Social navigation is also needed simply because search engines can’t keep up with the proliferation of content, including that which is dark, or behind firewalls. Then throw real-time into the gears…you need people intelligence and prioritization to assist the machines.3. Is it any wonder that so many SEM professionals dove headfirst into social optimization, which includes a huge leap into content publishing?
I like to think about the internet in terms of a library sometimes…things like Google are sim. to the card catalog…but no matter how good the card catalog is, you still need a Librarian to help you sometimes.So I think services like google will always need to exist (and will always be used)…but I also think there will always be a need for the ‘social’ aspect…
I think social beats search for a lot of things… certainly Twitter is a great source for interesting, useful, and amusing content. If my friends find it interesting so will I, right?However, I think the dishwasher example is illustrative. Who tweets about dishwashers? Almost no one. Incidentally, it’s a similar number to the people who LINK to great information about dishwashers (which is why Google is having trouble– it takes very little link building to rank well for dishwashers). I’ve recently remodeled my house and can attest that the farther Google gets away from the “linkerati” (influencers who link to things), the worse the results get. With all of my boolean-fu, I oftentimes could not find anything but adsense farms or sketchy looking commerce sites.I think likelier we’ll see Google start to incorporate social INTO search… But that still begs the question– in this new world of content generation, are people really going to take the time to generate content about their dishwashers or home remodeling efforts?Social will save us in the world of content discovery, but we’re still going to have to look to the Google search box to save us from SEO spam.
I figure we’ll see pretty profound changes as our conceptions of [online] authority evolve. Measuring links and references has been a powerful approach, but how about measuring the number of purchases a given dishwasher guide drives, or the resulting customer satisfaction levels?The amount of content online isn’t going to take a nosedive, which leaves us needing/relying on better filtering tools & techniques. Run all this content through your graph to filter out noise, and you’re left with a social signal; it’s definitely one valuable way to make the world smaller. But the proliferation of new types and sources of data and finer data granularity should fuel its own dimension of approaches.
Maybe dishwasher vendors should create incentives to share links socially
If incentivizing tweets becomes widespread, it will effectively become the same as offering an incentive for blog posts or links… Google will either wipe out the value of the links or penalize the person giving the links… in most cases not the dishwasher manufacturer.Also the people tweeting would have to add disclosure
I should have chosen my words more carefully. The volume of comments on this post has been hard to keep up with and I’ve been trying to weigh in with short burstsI don’t think its a good idea to incentivize useless tweetsBut if you buy a dishwasher, that’s a meaningful decision by you and if you can easily share that decision (like a foursquare chekin to a coffee bar) that’s a good thing and valuable to everyone if it becomes discoverable
Maybe the dishwasher should have a ‘tweet me’ button or even just a sticker saying ‘like me tweet me’
Or buy me tweet me
Absolutely. I agree. For last 15 months we developed semantic algorithms for automatic ranking of products based on texts of products’ reviews and specific technical parameters. Now we trace more than 1500 sites for reviews day by day. What is nice, as a result one can search for “good dishwasher” and can get good answer for that. We specialize in algorithms not content, as the content comes from social and non social networks, people like you like me. And it’s not giving you links like google, but it’s kind of snapshot – the most expected information by customers.
Just a few points:Crap is in the eyes of the beholder. What you get from Google depends very much on how you interact with Google and who you are (even without personalization of search results). Same with social networks and social discovery. Google’s search “algorithms” are based mostly on rather simple counting of social gestures – link vectors, link text, individual clickstreams and browsing histories, landing page bouncing rates, etc.Google is more social than it appears on the surface (I even suspect they prefer to sustain the myth of math/geek wizardry in the popular imagination).
lol…..sometimes mikey makes it too easy to pounce on him…..well it is the holidays so maybe mikey is just giving out the chanukah presents…..thanks mikeylet’s squash the beef on this one:1. believing automated content creation is necessarily inferior or unhealthy like fast food is a false assumption IMHO. automated content creation is getting better and better and some of the more advanced SEO folks can create automated content that gets comments and leads to a community. 2. the search engines have a lot data at their hands and some very bright folks working for htem. to think that they cannot sort out the crappy automated content from the good automated content is possible, and you could argue it’s working now, but i think they have the assets needed to filter out a lot of crap and will continue to get better at it. the comment fred left on mikey’s gossip rag and the extremely controversial, antagonistic headline of this post illustrate how search engines can filter out the crap — by leveraging social data, particularly the highly coveted social graph, which all search engines can do to some extent via their email apps, and probably other ways.
I tend to agree with the kid on this one. Content creation for SEO purposes, automated or otherwise, isn’t necessarily a bad thing. 1. The content often has at least some utility and value, and 2. it can serve as an entrance into other useful content and services.Will the content be as good as something that you or the rest of my social graph would recommend to me? No, but if I’m the first in my network to research something – or it’s something I want to keep private – I may not have that luxury. The search engines we end up using will be the ones that do a good job culling the cruft.I realize that you guys are primarily talking about content farms, and there is certainly a lot of spam out there, but at the same time, if you’re building something around a vertical, it is essential to have a content strategy, perhaps primarily focused on social and sharing, but also on keyword research and SEO.
well kid, i thought you are going to say that the piles of automated content hide the truth that will set us free? search can be good for facts, but for commenting? interpretation? identifying trends? insight?
the only thing that hides teh truth that sets us free is people — whether it is google engineers hiding 9/11 truth or amateurs like mikey endorsing big media candidates and not talking about real issues like the threat of the cybersecurity act. that is why i pounce on mikey, because he is an amateur who makes money off providing a journalistic service but does not hold himself to the standard of a professional journalist. of course, very few people do, and if mikey was a real journalist, the people would go to someone else who can coddle them with lies. technology only goes so far. all this internet stuff runs into a dead end until its users are mature enough to see and pursue its full potential.
Seriously. To think that innovation won’t help sift through and surface quality content is pretty short-sighted.To me, it sounds just like a “back in my day, we walked 5 miles to school in the snow” kinda complaint; we tend to think that things were better in the past, and that people don’t think, or care, etc. the same way they used to. And they don’t; but that’s something only the “old / incumbent / disrupted / unable to adapt” complain about.
You might be right – but, if spammers found the way to created worthless content – then why will they not find a way to create worthless soical media? Will our friends and trusted sources end up being spammers as well?
How do you spam yourself into a social graph?
pretend to earn someone’s trust so that you can betray (i.e. spam) them later. forum owners and CIA operatives see it all the time. i bet wikipedia’s governance is heavily compromised by spammers.
As a follow up, this morning I heard on the news that Kim Kardashian gets $10,000 for every tweet that mentions a sponsor. So, the question is, does she actually like these products and endorses them due to liking then simply gets money for it or is it just a way to put money in her pocket – regardless if she likes the sponsor or not? If she doesn’t – is this spam?
PageRank’s < a > tags were once enough to guesstimate online reputation of content, but no longer. I agree that today’s approaches will be augmented by an overlay of “trust” via your social graph and/or other reputation systems, for example the click-rate of your bit.ly links or scoring of amazon book reviews or slashdot comments, etc. Curious to know how many degrees of separation preserve quality and relevance, without creating insularity/diversity problems. I’d guess 3 or 4.
Problem with link click count is it can and has and will be gamed. Doesn’t mean we shouldn’t use it because it still is a great signal but its been gamed
Yes I was also thinking how easily reputation systems are gamed after I posted. Perhaps other counters to my first comment are that the majority of internet users are still fairly passive, rather than active, nor is there an effective solution to identity yet (also fraught with privacy problems).Nice to see you, Jarvis, Dixon and Arrington writing about the issue. Some interesting thoughts and it really feels like we’re about to enter a new phase.
I hope so
I keep up the assertion since last year that Google is a social network that is there when you need it, and not when you don’t. This makes the data asset immune to classification as a “social network” and therefore immune to whenif the fads of today pass. This also primes them (Google) to survive the service fatigue that can and will happen elsewhere.
“Google is a social network” – absolutely!
Yep – currently it’s a social network based on the graph of webmasters and other publishers – with links as the signal.Future versions will include everyone in the social web – with relationships, comments, RTs, and implicit behavior as the new, expanded signals.
You mean PeopleRank… 🙂
Very, very well put. Quality matters, and content creators who build solid relationships with their readers in the social web are the ones who will see the fastest growth moving forward.
Great post Fred…as I think about how to build truly big media businesses in the future – at the end of the day you still need to build a brand. A brand that folks can direct nav into (still the most important metric right). I think the social graph can go a long way to help brands establish toeholds, just as the SEO approach gets trial. Do you think most people, or put another way, an at-scale audience, for a media brand can be maintained/developed through social? Clearly for sites like your blog, or Techmeme, it’s all there. But if you’re really trying to create a big business i think you need to create a brand that people get and trust and will go to. What’s happening is by focusing on SEO everyone is borrowing brand equity from Google. On the social side, it’s not so easy. You can join Twitter, but that alone won’t help you. It takes a fair amount of work to create these uber filtered experiences. In general, I am skeptical that large audiences that can support large media businesses can be developed with social as a number 1 channel. Of course, the exceptions are the FourSquares and other inherently social or technologically-savvy audience-targeted services for now. It will be interesting to see it plays out. The other question/implication is, if you’re right, all new services MUST BE social to succeed.
Well facebook is closing in on 400mm people worldwide and is opening up. Look at what our friend mark built on facebookI think it can be done
for all the newspaper bashing that goes on today, this is ironically one place we can perhaps learn from the papered past. newspapers were, in essence, social graphs. content was created-selected-recommended-approved by people with a level of trust, expertise and credibility. i subscribed to — “followed” — the newspapers that matched the ideals important to me. instead of a single editor-in-chief, our social networks now act as a collective digital intelligence. the role of the editor is now part of the meritocracy.
Yes but now your wife or son or college buddy can be one of your editors
That might not be a bad thing – that’s typically your trusted circle. But if it is, well, now we have all sorts of fancy privacy controls. And, to be clear, i’m using the word ‘editor’ to mean filter. That’s what newspapers did: they gave us filtered versions of news and perspective. And that’s what we create for ourselves with our Twitter lists.
Incentive to deviate: that’s one of the few things I remember from game theory at Yale (Barry Nalebuff, Fiona Scott Morton, Ben Polak). Content farms (and spam in general) deviate because it pays off. So either we kill those incentives – and the case is loud and clear why google should – or we create incentives that pay better. At Ripple100 we’re betting on the latter. The interesting point about social graphs and filters is that incentives aren’t financial.
Read yochai benkler on that last point
As a point of Fred-ish corroboration, after my bleat about the awfulness of Google search in finding dishwasher information, among the more interesting side-effects has been all the people in my social graph who have emailed me useful information on dishwashers. This social-+-email thing could be huge — huge I tell you.
The problem with the social+email that you suggest is that you are still left to distill the data sent in by the elements on your social graph to arrive at some meaningful conclusion. If that social net is large then you may get inundated with data. The key will be to somehow set constraints are to what you are willing to accept as you broadcast to your social graph and hope that they read them and also filter out any data that does not meet your criterion of an acceptable item, “dishwasher” in this case.
I like the european ones myself. They are smaller but quieter and very nicely arranged inside
There’s possibly an even greater benefit to “social search” depending on how it’s done. It the search is “human powered” (i.e., non-algorithmic, like Aardvark for example) the results should be a much richer reflection of ones own tastes, preferences, likes/dislikes (presuming your friends list actually know enough about you!)For that reason social search is better suited to questions with more subjective criteria and/or criteria that’s difficult to scrape from content. For example, quality of customer service, music recommendations, entertainment (restaurant/movies). Or dating – see Thread.com.
Google will find a way to detect fast food content and lower its rank. We’ll soon see some kind of Google initiative that takes source of origin into account. This is actually a good thing, search engines should be ranking original sources higher than both farms and blogs that lift content (Huffington Post, for example). The rise of content farms will only make this a higher priority for Google.
Regarding finding the ‘source of origin’ for content – there’s some fascinating possibilities in this paper from Cornell: ‘Meme-Tracking & the Dynamics of the News Cycle” (PDF) http://bit.ly/WA1tS
+1: social is the next step for search. i think ‘realtime search’ is mostly smoke and mirrors with a side dish of scaling and algorithm tweaks (ex. how long did it take google to implement real-time search once they cut the deal with twitter?). but social will redefine and disrupt the search space. it will require fundamentally new systems and UEs.now if we take social becoming part of search as a given….the idea you touched on of content ‘pull vs push’ becomes very, very interesting. when will i find information, and when will information find me (before i even know i wanted it)? ….a real mindbender if you really go deep into it.
Couldn’t agree with you more. “Academia at large” (Wed 2.0 and beyond) will evolve to a more robust level. We naturally migrate toward what works. That comes from honest debate,dialogue and testing. Crap will get tossed out.
I’ve been harping about this to some colleagues and friends during the past few months. In order to be relevant in terms of size and SEO, a compromise in content quality is usually required. Demand Media has built the majority of its business model around this idea with ehow and expertvillage, which are now under one domain. Wired Magazine wrote an article (http://bit.ly/DemandMedia) about their methodology to algorithmically determine what content to produce (text and video) in massive scale all based around SEO tools. Demand is far from being unique in this sense… there is a whole universe of other SEO driven sites. Unfortunately, search is largely predicated around the frequency and quantity in which a site can produce pages and not well thought, deep information.A little off topic, but driving higher click-through rates on google ad sense has also led to intentionally lower content quality. If you can win the SEO battle on specific keywords, present low quality content so the user is still in “search” mode, which ultimately drives a click-thru on the ad sense on page, more revenue.
UPDATE: Just realized Arrington’s article linked to the same Wired article that I mentioned. Complete coincidence, but at least we’re consistent.
For the most important, high value purchase decisions, I think that Social + Search can be a 1+1 = 3 scenario. I’m in market for a high end TV, and I recently asked my Facebook friends for recommendations. I received 15 great comments on plasma vs LCD, screen size and resolution, etc. Now that I have extremely valuable, trustworthy feedback from my Social network (product reviews), I’m empowered to go Search (purchase) for a TV with confidence (and I’m more likely to purchase because I’m not fearful of spending my money unwisely on the wrong TV). My search is improved because I now know exactly what I want, and the great product search tools out there will empower me to make an efficient purchase based on focused comparison shopping.This is just one example of a scenario where Social adds value to Search, and I’m a happy consumer and big winner
Curious where you see Maholo in the mix? As much as my friends are the “first order” (perhaps intersecting with neighbors who more often add better local value), what about the “old days view that subject experts can be powered to organize search results”? At a minimum it feels like it’s adding more value than brute force SEO content that contributes to the problem and provides little curative value.
The quality of information obtained today is held hostage to one’s ability to drink from a firehose.S v S is simply an exercise in selecting which firehose to drink from. The good news is that both project a healthy stream. The bad news is that there may be additives in the water.I was recently looking for a specific arcane waterproofing product. Got it from the Internet at large — search.I was looking for a good contractor to install it. Got it from the neighborhood ListServ — social.The long term answer is going to be BOTH.
I would simply search, for JLM
Most people’s trusted source is a machine. Google is the #1 brand in the world. It continues to get better and more relevant b/c the amount of data it’s crunching (cross-platform too) is astonishing. Google is also going to more personalized results sets and this will surely help deliver more relevant SERPs and make Spam less of an issue. Ultimately, content value is a user experience questions that data will answer. Social graphs do not have the data scale and feedback loops associated with them to help get the most relevant piece of content in front of someone the moment they want it or need it.
The problem is that these fast food content providers have exploited our current measures of quality. If we leverage the social graph as a way of assigning quality ratings that are tied to people, not sites, we’d have a new and more effective way of rating content. The current social network services are great for passively sourcing content, but not as useful for search. I’m not going to have an easy time searching for a dishwasher recommendation from my social network. We need a new service that does Friend of a Friend rating and filtering of content.TweetMeme + a new flavor of PageRank, call it FriendRank, that considers your own relationship with those who retweet. I think I have a new project now.
The challenge is the personalization and recommendation of content and ADs. That will not be based on “static” keywords but on dynamic user demographics and “interest fingerprints”. We will not market against keywords, but against (for!) specific users. Just by continuously looking at social graph, passed-links, location, language, etc we will tell a lot about someone. And we must be careful to serve them well and with honesty.
Another thought that I’ll put out there for anyone interested: it’s worthwhile keeping an eye on what Doc Searls and the VRM (vendor relationship management) crew (http://projectvrm.org) are doing.While VRM is focused on commercial transactions rather than content discovery, I think that there’s a pretty significant overlap in the issues involved and the tools that will be required.
As Fred has alluded to before with the “golden triangle” once the super users (such as these readers) harness mobile, social, and realtime into a formula digestible for the general user to understand. People will begin to look past this generic content and begin forming strong social graphs, focused less on volume of followers/content and instead on quality as it will be easier to realize the benefits/value. It will be interesting to see if some people move towards paid content to help find their selected material. To the common user the Google search box organizes all this crazy content as they’re still unaware of the strong user communities and social networking tools to help streamline the massive amount of content being created daily. Once these communities and tools reach critical mass they will be a welcome addition to general users opening up a world most of us more advanced users see everyday in comments sections such as these.
Social Media designed to enable individuals to voice their opinions as loudly as Mass Media tastemakers may not be so bullet proof. Jeff Bewkes’s memo to the staff on the day Time Warner officially split with AOL implies that despite an abundance of choice, blockbusters continue to be the most popular and garner the lion’s share of consumer media spending. In other words, in the David v. Goliath strategic game, Goliath is capitalizing on the “divide and conquer” advantage.I assume you are familiar with Doc Searls’ related post: “The Revolution Will Not be Intermediated” http://bit.ly/5m9L3h triggered by the same posts by Arrington and MacManus. I agree that “new mass media” will not be the “intermediary”, but revolutions do have leaders. Leaders whose authority emerges organically with the development of a community that balances the diverse interests of individuals.The void in the marketplace is technology and media designed to enable or facilitate “organically grown” leaders, and the communities that empower them, to organize.Katherine Warman [email protected]
100%Though I also question it as well. I was nearly done with Clay Shirky’s book before I had to pack it, and I need to wonder about what actual social graphs look like in three dimensions. It isn’t clear to me that any of this is going to make any sense until we graph out in multiple dimensions. And that’s why we are all in trouble- how do you conceptualize this. If you had to make this visual? This is so difficult.
You’ve got the challenge right – how do you visualize multi-dimensional solutions?But, ironically, we are in a position where the “buyers” see the value immediately. It is the technology vendors who do not see the value to them of partnering with other best-in-class technologies to execute this multi-dimensional solution.The buyers get the value because we are grounded in the real world. We conceived our multi-dimensional solution by using real world live events as the model. Real world live events are inherently multi-dimensional in terms of timing (pre-promotion, preparation, live production) diversity (value to the audience/performers/producer/brand sponsors), and impact (the benefits of the camaraderie experienced at an event to deepening and sustaining community). Our buyers immediately see the benefits of our solution to network, organize, and maintain the integrity of their people, data, and sense of community.But technology partners which are really good at one of these dimensions, seem to be reluctant to see the value of collaborating with other technologies which are good at one of the others. This is not a financial issue – we are offering money on day one, not the “hope to make it big on the back-end” proposition.Anyone have any thoughts on how to overcome this hurdle?K-
Great post. My new social search company, Funnelscope (www.funnelscope.com), is looking to solve this problem for hotel search. We programmatically aggregate all the content but we overlay it with a social framework for users to incorporate input from friends. So you get personalized hotel rankings based on a hybrid model of algorithms and social input. I think it’s too difficult for humans to separate out which to leverage at the appropriate time and think companies need to build more robust solutions to account for this issue.
Yes, social beats search an this is the big challenge for Google. If, as looks likely, search habits can be captured by either the owner of the public conversation and the influence hierarchy or the owner of private conversation, attention, the social graph and brand affinities (who is desperately trying to flush more private conversation out into the open) Google could take a real hit. Handicapping between Twitter and Facebook I’d have to give the edge to FB. They already have incredible targeting capabilities for display — a meaningful capture of real-time intent would make for a compelling value prop to marketers. I’m sure we will soon see significant search innovation (extra credit for alliteration?) coming from them, with or without Bing…
What about the “semi-social” graph? Here’s what I mean by this:Google has done a great job getting spam out of inboxes. Yes, this has a lot to do with various checks of header info in the email, originating IP address etc. But a key part to being great at spam filtering is learning from the people who flag spam in the inbox.This doesn’t require a direct connection between me and you: if you flag something as spam in your inbox, it might well help keep such an email out of my own inbox.Google has added a similar layer of flagging & rating to search results, which I imagine is a first step.So I don’t think it’s social vs. non-social, there are variations in between which will be very useful, too.
yes, the way email filters get the spam out is a combination of algorithmsand people flagging spamit seems that the most powerful solutions require a combination of human andmachines
At the Singularity Summit, Peter Thiel said this is one of his primary areas of investment interest going forward. It was a combination of human and machines that saved PayPal from collapse back in the day
Social technology is important because the human, “real world” dimension is the “missing link”.The next step is to recognize the need for sense of community to maintain integrity between individuals.There will be many “phony” communities that will fail because they lack the compelling bond of “real world” communities: when each individual feels they need each other to thrive. Greg Brown (the singer) at a concert shared an example in which that bond is even more compelling than thriving. In Alaska the bond is a question of survival. Imagine what a community is like when you know that any one person could be driving the one car that comes along when you are stranded with a dead battery in below zero temperatures!K-
Sure, although community will mean many things to many people. Community also doesn’t need any element of time or proximity to matter. But to your point, social technology is hugely compelling. In my mind, Facebook Connect (or something like it) is the foundation of this future
A total sidebar to this topic is that it amazes me how people struggle to grok the difference between (the goodness of) crowd-sourcing as a mechanism for amplifying the good stuff, and filtering out the crap, and user-generated content, which is fairly/unfairly perceived as amateurish. Amazon, as a product discovery mechanism, speaks to the power of the crowd-sourcing model in action.
it’s just hard to digest that part, if not the major part, of any consumption – food, tv or info, is crap. But this is how it is.
I agree with this point. Semantic search, structured data, and filters will get me there. I dont need my social network to answer this for me – I can use Yelp. I can use Hunch. As in, it doesnt have to come from my friends, or people I trust, it just has to come from lots of strangers. The headline on Facebook ‘Beacon’ was that people felt their own privacy was invaded, but what you would find if the feature had stuck around longer is that the benefit of a trusted review is not worth the cost of diluting a true ‘social network’ with marketing and product references.
@MichaelMuse, like you, I put a lot more weight on ‘like minds’ than friends. Personally, there is something liberating about reading a bunch of opinions on a product, checking the composite opinions, filtered on different parameters and then cross-comparing with your own priorities.It seems clear that there is a lot of room for innovation in that area. I like what Hunch is doing although I found their mechanical turk demo app pretty disappointing. 2.0 will be better I am sure.As to Facebook, and Beacon, part of what got them flack is the general heavy handed/sneakiness of presenting major strategic changes without truly soliciting their user base. A second part of that is not defaulting to the crowd over friends, which feels like a much better model, and also a secondary way of creating more loose ties, so as to grow your overall networkIf interested, here is a recent post I wrote for O’Reilly on Facebook’s new “privacy” policies:Is Facebook a Brand that You Can Trust? http://bit.ly/8xySuWCheers,Mark
just another factor which often overlooked: TIME. We work more, we have less time, and this mere fact shapes our content consumption. This is also a psychological effect: the more options and opportunities you have, the shorter your time seems. And who doesn’t want to be a musician, writer, photographer, cook sushi, a scientist, lands discoverer and entrepreneur, all in 5 steps easy steps?Quality requires time, and we make the quantity/quality mix all the time. Search is on the quantity side, social graph is the quality.
In the interest of clarity, it would be good if you could give an “operational definition” of what you consider to be a “social website” and what you consider to be a search engine. For example, what would you call birds.com and/or food.com? Social? Search? Something else?Of course what’s important is not simply someone’s OPINION, but rather a clear description — such that the classification of “social” vs. “search” (vs. “something else”) could be replicated.Otherwise, we might just as well talk about unicorns.:) nmw
Like you said, I think that headline is a bit dramatic – “beats full web search” maybe. I think we’re about to see a huge boom in “socially culled” search.
Long live the age of content curation. As more and more content floods the Internet, people will increasingly look to trusted resources to make sense of the rising tide. One’s social graph will play a critical role in navigating the ever-growing stream of content.Trusted media institutions like The Wall Street Journal, The NY Times, and Newsweek will evolve into multi-faceted organizations who will spend as much time curating outside content as creating their own.
“Trusted media institutions like The Wall Street Journal, The NY Times, and Newsweek will evolve into multi-faceted organizations who will spend as much time curating outside content as creating their own.”Now that’s a fascinating idea. There’s still a lot of value in the editor/curator functions of media institutions, it’s their production & monetization model that’s shot. The question is will these behemoths be able alter course quickly enough to capitalize on the value before their trust and influence dissipates?
It would be difficult, if not impossible, to lose of all the trust these institutions have built over the course of many decades. Rupert Murdoch would not have purchased The Wall street Journal had he felt the WSJ brand could be diminished.It’s critical, however, that the leaders of these companies focus upon quality over quantity. They must ignore the short-term gains reaped by organizations like Demand Media, and instead focus upon the long-term benefits of maintaining their positions at the top of the media pyramid.
Yes, and in fact these companies and others have been vetting stringers and freelance journalists and photographers for over a century-weeding the wheat from the stalk so build up a reputation of quality. As oter commenters have said, these business have been and are in a postion to meet demand for quality and trusted content, but they are dragged down by there old business models
As these organizations free themselves from the weight of their legacy costs, they will emerge as lean enterprises tasked with ensuring that consumers receive news that’s fit to digest. Their long histories as monoliths of traditional media will give way to futures shaped by reverence to the consumer. They will create, curate, distribute, and analyze content across the globe.
I think a the yoke of the old business model will be too much burden for many of the dinasaurs too survive. They have given too much window of oppertunity to the many new VC backed media companies that are poing holes in the old models. Of course, we have seen alot of flame-outs of large VC money invested in new media models, but eventually some will get it right.
I think for most companies we need to treat the two as equals. You can see more immediate results from social media, but at the end of the day there is nothing more powerful then customers coming in the front door without your having to do anything…ie organic search.
When ‘social’ is being discussed, people end up referring to either the FB social graph / twitter. However, having a popular blog helps as well – write a post asking for a recommendation / suggestion and you’ll have several good responses.Blogs maybe not the ‘in’ thing anymore – but having blogged for 3+ years now, I continue to reap benefits from blogging regularly – from my loyal readers and the relationships i’ve built with them over the years.
This is great news for those of us who work hard to create good content then as people rely more in networks to discern what is crap and what is not.
Most of my best discoveries in life have been totally accidental.Let’s not make discovery too much of a science …
this is the smartest post i’ve read all week.
The worst part about Google spam is that the spam pages are populated with Google’s Ad sense links. Google is usually the company making advertising $ on the next click out of the spam page. So if the don’t get you with a paid result from a search, they get you in the natural results by running their ads on the Spam pages. People don’t realize how much of ggoles revenue is from ad sense, not adwords. I wonder how aggressive they really are about removing these pages given that they are a revenue source?
Fred, I agree with most of what you say on this blog, but not this post. How is creating content around what people are looking for bad? Isn’t that giving the market what it wants? And if it’s lousy content, it shouldn’t rank. If it’s lousy and it still ranks, it means A) that it’s so far down the tail that nothing better exists (so like Fast Food, it’s still meeting a need); or B) Google is not working as expected.And as far as social beating search… in theory it should. But there aren’t enough content creators for social to provide enough answers to most people. Maybe the digerati can get the occasional answer from their networks, but nobody else can. We are years away from the content producer to content consumer ratio getting high enough for social to be a viable competitor to machine driven search.
Bingo Fred. “social tools will allow us to decide what is crap and what is not. our social graphs will help us. search engines won’t. it’s a lot harder to spam yourself into a social graph.”I would add the following to your statement – “this new filter of social also turn upside down two big elements of online advertising – behavioral and contextual of which both are different in social”; I’d also add that that notion of relevance is still elusive at this point (in terms of scale). In all it’s wide open for investors and entrepreneurs.
“It is harder to spam into social graph” True but will you always do what your social graph suggest or does ? If I understand it correctly , do you mean to say that next time I want to find some relevant piece of article , some tutorial , some piece of news I would go to my social graph ?It assumes that the whatever info I want in my life some else in my social graph should have knowledge of it.Agreed that companies like Demand Media are spamming the search engine but I guess search engine will outsmart them.The future will not be owned by newspaper and media . It will also not belong to search engine. It will be a mix of all.
automated internet machines easy for everyone else, yes. because you shouldn’t need an internet marketing degree to build an internet business. and it can’t just be search optimized, internet has people and places.in 2010 it should build for search, social, mobile, AR, internet everything.
It’s called Astroturfing or Farmville-lobbying if we have to beleive the latest trend — and it’s bad, and it’s here to stay if we beleive Jon Zittrain. Given how “patrons” have operated continuously in Italty for millenia (from Ancient Rome to contemporary camera-phone enforced vote-buying) I’d say we are in for a long run.
it’s been said before, but “search” is not the problem—because it implies that “to find” is the goal. sometimes, sure, but nearly always it’s something more. here’s a set of categories adopted from the ed-psych literature: find, understand, apply, analyze, evaluate, create. can machines help? sure. can social help? sure. sorry, what was the original question?
I would say, with the help of machines, communities, and trusted sources can form a social graphs to make content spamming much harder than the current search model of finding information.
Couldn’t agree with you more. We focus on entertainment content, which as you know, is almost exclusively shared across the social graph. Great suggestion engines notwithstanding (they are getting much better) — a normal human being just does not go into a search box and enter “bring me something that will make me laugh” — yet.The blending of suggestion and semantic logic in discovery works best when it is the filter on top of one’s social graph. Our EyeTMedia Ad Intelligence platform brings these factors into an ad world that is, at least for the moment, obsessed with content as the pivotal factor, as well as crude demographic and other old world context such as referring page, pixels and cookies. We transcend (such a high minded way to say “we have improved”) this by exposing users to mavens and passionate enthusiasts, careful curators and freaks in their social graph.Discovering a whole person and their unique (if somewhat skewed) perspective on the world and on entertainment content (and all content for that matter) is a faster and more exciting way to trawl for new content discoveries, and a more potent way to explore new corridors, getting lost in content strings of friends-of-friends. Our end of this game is to measure what works in terms of ad performance predictions, and in that we are white-hat, good-guy behavioralists and contextual folks. We have lots of other goodies and features that thrill publishers and consumers, and provide extraordinary value to marketers. But that’s on the way to ad performance measurement too.Back to your point, search spam and content factories are merely there because they can lure people to click on them. If we put such blend of suggestion-engine and semantic logic on top of social (referral/recommendation) discovery, things sort themselves out quite rapidly.I remember in 2003 at Yahoo! I was inside their flailing small business unit talking to the business guys about how to improve the way they got the attention of small business owners. The one thing I called-out as a short-term, easy win was what I referred to in those days as search results-spam. Mind you, others are much more well-acquainted with this than I am, But at the time, it seemed like a simple way to explain why small business owners were fed-up with search as a quick way to find services and goods they needed, and why they did not trust search enough to jump into the Yahoo! branded small business sites and buy ads themselves. Yet nobody at Y! wanted to talk about it. It was as though this was the crazy old uncle in the attic apartment.So this is a frustrating old problem, made worse by the fact that the devices and mechanisms to fool people into clicking are more sophisticated and more insidious.I think that if it does not get fixed fast, we may see our kids shifting to the concept of a Balkanization of the Web that limits them to the “Web of their friends” as the more private, more calm and quiet Web that meets their direct needs. From there they *may* explore and launch-out, but probably only in the context of recommendations from others in their graph who were pioneers into the void and who did the hard work of sorting and sifting for them. Beyond “crowd sourcing” this goes to friend-sourcing and social-graph sourcing that infers a bit of dependence on reputation trust, a bit of easy-button convenience and a very interesting implication on advertising and invention.Speaking of discoveries… would you be able to post a cook-book on how to get a “radio” like the one on your page? -thx -Matt
“I love my computer. It’s where all my friends live!”That being said, we are online socially and the need for recommendations by friends or non-spam authority is key. Facebook and twitter can be great at this but I see this area as undeveloped territory. Hmm a social search engine based on what your friends are liking would be a pretty bad azz service. But I think the trick to developing it would be usability and making it’s aggregation seamless if not invisible. I’d enjoy being able to search for “marketing tips” and having an aggregation algorithm refined based on searches/favorites by Seth Godin, Steve Jobs and my personal pals in marketing.
I think that social media might probably be the best way to discover current content but search will dominate long-tail content – having said that, both will have their spamming challenges. So perhaps a third way of differentiating good from bad content could be based on tracking implicit user behavior and how users interact with content (for example what Tynt is doing with tracking copy / paste actions). Full argument here: ttp://www.wmediaventures.co…
I don’t think that search gets replaced. Rather, existing search algorithms need to be augmented/improved with data from the social realm. When your tweets get retweeted extensively, that’s a real world indicator that you are trusted. Today those “links” don’t really help you in Google (as far as I understand SEO). We (at Oyster.com) get mentioned by hotels on their fan pages in Facebook – those links from the hotel to us (but within facebook) don’t really count for much in Google though clearly they should. A lot of people find it easier and more appropriate to link from FB than they do from a web site.
Throughout history, Trust is something that is usually earned over many years. Retweeting measurements are fly by night measurements lasting a few weeks or years at most- In fact Retweeting is more like Rehearding. If an Italian Bostonian named Ponzi were still alive, he would have a field day with Retweeting.I recognize that retweeting has huge value in capturing the current zietgietz, but the social group algorithms need to also include, perhaps in some weighted fashion, the tons of information that have been filtered and vetted through a thousand years of history rather than just a few weeks of pop culturedays/weeks/moths/years of history.I further propose that one can live an equally successful life, learning off grid by learning from sources less vetted by current pop culture and more vetted by 1,000 years of historical filters. Aftern all, this is the type of information that shaped Pascal, Hume, Ben Franklin, Thom Jefferson, Abe liNCOLN, tHOM eDISON (history’s greatest entrepreneur and Greatest VC!), and many more
Talking about these issues very much reminds me of the rising french start-up #pearltrees enabling users to save and share their interests thanks to a very innovating visual – the guys there do a great job and it’s worth discovering (their web site is http://www.pearltrees.com and you can get a hint of what it’s like at http://is.gd/5qsbm )
As someone who has been working on a human-powered alternative to search for several years, I’m happy to see this increasing recognition that uncurated search engines are a crude tool for finding information online. While social tools may take their place for well-connected people, the vast majority of Internet users will turn to trusted third-party sources that consistently deliver important, relevant, reliable and comprehensive information, from a variety of resources across the Web. And the only sites that will do this well are those that rely heavily on a human touch. They will inherit the position at the top of the Web food chain that search engines abdicate. As I explain in this blog post, search is dead, long live gourmet content.http://foundingdulcinea.blo…
Perhaps, but if you know what you are looking for, that seems to indicate that you have already looked at the subject, you may even have a specific site that you go to, if you already know what you are looking for.ex: if I know if I am looking for a variety of opinions of vc financing, i might go to thefunded.com and a few other sites. If I want to get current information on the tech and vc industries, i know to go to avc.com and a few other blogs.f I want to get Boston weather or local sports info, i go to boston.com. I never do search for these areas cause I have already invested the time search to arrive at these few sites that satisfy my info quest the best solutions for me. I dob’t have time to continuously search for new sites on these subjects.However, for areas that I don’t know, such as how to fix a dishwasher, i will do search and weed thru a number of sites to get an overall feelinging for the best answer.
I don’t think we’ve seen social commerce done right yetWe will
Agreed … This space is ripe and represents a HUGE opportunity to fill a need for social discovery.