In Search of a Better Algorithm

Try searching on “allen iverson email” in Google. The third result is a link to this blog.

Why?  Because I once posted the fact that I love the way Allen Iverson plays basketball, because the word “email” is high up on the front page of this blog, and because there are a ton of inbound and outbound links to this blog.  But is AVC one of the best results when you are searching for Allen Iverson’s email address?  Not likely.  But there are close to a couple dozen comments on that Allen Iverson post that suggest some people think it is.

And the same thing happens to lots of other bloggers.  Try searching on Google or Yahoo! for “oprah backlash james frey”.  You will be directed to Brad Feld’s blog for similar reasons.

Why am I telling you all this?  Because as great an experience as searching the Internet is on Yahoo!, Google, Microsoft, or Ask, Internet search is still a very primitive technology.  Rarely is the first result of a search the best result for my needs, regardless of what engine I use.

I’ve been thinking a lot about search lately.  I do a lot of searching on the Internet even though I have literally hundreds of sites bookmarked and have at least fifty to a hundred sties that I visit on a regular basis and know the URL by heart.

I tend to use Yahoo! for most of my searches as I have made it the default search in Firefox.  After that I use Google.  I rarely use Microsoft or Ask.

But last week, as a result of the relaunch of Ask.com, I did some searching on Ask and I got very different results on my standard test searches; fred wilson, vc, union square ventures, wilco, flaming lips, digital camera, and a few others. Ask does not appear to be using link ranking nearly as much as Google and Yahoo!

When you search on “fred wilson” or “vc” on Yahoo! or Google, this blog is the first result on both for those search terms.  I have always thought that was because those keywords appear high up on the front page of my blog (vc is in the title) and because of the large number of inbound and outbound links that this blog has accumulated in the 2 ½ years that I have been blogging.

But when you search on “fred Wilson” or “vc” on Ask.com, you get a whole bunch of other results. It’s basically what those search terms returned on Google or Yahoo! back in 2003 before I started blogging.  So that means to me that Ask.com doesn’t seem to care much about link rank.

I don’t know if that’s a good thing or a bad thing. Because when you are searching for Fred Wilson, there is as good a chance you want the artist, chess master, or rock n roll band, as you want this blog. When you search on “vc” on Google or Yahoo!, your first link is this blog.  Is that the best result for vc?  I doubt it.

Text search works well enough to be useful, but it doesn’t work well.

And before we get close to perfecting text search, we are off to new horizons with audio search, video search, etc.  Will Google, Yahoo!, Microsoft, Ask and others continue to invest in improving text search or move their efforts to searching other forms of media?  I suppose the answer is both but clearly there is an impression among many that text search has “been done” and that impression is wrong.

I believe there is opportunity in improving text search, but few investors want to take on Google, Yahoo!, Microsoft, and others by backing another search company for good reasons.

But the immaturity of search was one of the many reasons I loved our investment in delicious so much.  I thought, and still do to some degree now that Yahoo! owns delicious, that searching user generated tags instead of (or more likely in addition to) some computer generated index would generate a better result.  Of course, tagging must become a much more popular behavior before we can have a tag database that can deliver high quality results for all search terms.

There is also the promise of shared searching which Yahoo! is promoting with MyWeb (why don’t they just merge delicious and MyWeb and let us choose either interface?).  David Hayden is chasing a similar vision of shared search with Jeteye.

And there are next generation tagging services coming like Plum that may offer some new ideas in this area of shared search and discovery.

I frankly think that an orthogonal attack on search via something that is seen as very different is the more intelligent way to approach this problem.  First, it’s more likely to obtain investment.  Second, most users aren’t going to start searching with a different engine unless they see the benefits first.  So you have to hook them on something where the initial value proposition is something else (tagging, social networking, looking at videos, etc).

And then there is Alexa to consider.  Amazon has supposedly opened up the Alexa search service so that others can use it.  I know of at least one company that has taken them up on that offer, although they found that the demand to use Alexa was larger than Amazon could initially support.  I haven’t checked back with that company to see how they are doing with Alexa.

What if we had an “open source” search engine that everyone working in and around the area of search could plug into?  The companies working in tagging, shared searching, audio and video search could offer their results/indexes to the open source search engine so that their meta data could be considered in preparing the best results?  Could this work? And what would the business models be for the companies supplying the meta data?  And would consumers adopt such an engine?

I am not sure, but I am sure that we are in the first or second inning of the search ballgame, and nowhere near the seventh inning stretch.  So for all those entrepreneurs who are way smarter about this stuff than I am, let me encourage you to be thinking about search as much as I am these days.  It’s still a huge opportunity.