Searching for something new?

Search is key to the popularity and success of the internet, this much is certain. Could you even imagine an internet without search? But as more and more content is added the task facing search engines become more and more difficult and they need to adapt.

On the surface search engines don’t look to have changed much over the years, inside they are rapidly evolving almost like a virus, tweaking it shape and structure rapidly to adapting to changing environment outside. Generally though they have been moving along the same old tracks, using vast databases to store details of information on the internet and complex maths to rank those results into some sort of order.

The idea of a semantic (meaningful) web and semantic searching has been around for a while. The exact definition of semantic of course changes from provider to provider, but its usually taken to mean getting clear, concise disambiguous information.

Most of the big providers have been chasing semantics like it was the holy grail. Google, Yahoo and Microsoft have all been racing to get something out first.

There seem to be a few common methods for approaching semantics; yahoo are asking content providers to include ‘specific semantic’ data within their pages that their search technology can pick up and work with. The other approach is to creating a database of objects like people, places, events then creating structured relationships between them. This approach currently both labour intensive and slow as there is not yet away to extract this for the web.

According to an article in read write web, Google who have dominated the search market for the past decade have already introduced some semantic technology into their search engine. If you ask “how old is Harrison ford” you will get the answer “Harrison Ford — Age: 66 years (born 13 July 1942) – according to http://www.netglimse.com/celebs/pages/harrison_ford/index.shtml” This currently only seems to work for a small number of searches, largely based around public personalities.

There are many other fantastic projects out there like twittersearch , trueknowledge, Haika, clusty, that work on everything from natural language searches and real time live searches to visual and clustered results.

One of the most exciting projects to be revealed recently is Stephen Wolfram’s Wolfram|Alpha. This takes an entirely different approach to knowledge management. Wolfram has created something he calls a ‘knowledge engine’ that will actually compute results using a huge system, with trillions of pieces of curated data and millions of lines of algorithms. It’s amazing that a project as advanced as the Wolfram project managed to remain so quite, for so long, with so many people working on so much data.

The site is due to go live in May 2009, Wolfram is quoted as having said “I wasn’t at all sure it was going to work. But I’m happy to say that with a mixture of many clever algorithms and heuristics, lots of linguistic discovery and linguistic curation, and what probably amount to some serious theoretical breakthroughs, we’re actually managing to make it work.”

Its obvious that we could be on the brink of a paradigm shift in search technology, but which way it will go, and how that will affect the way we use the internet is still to be determined.

Sources: Nova Spivack’s twine , Wolfram blog , Read Write Web blog , Smartsearching Presentation

References: Search Technology, Search history Privacy

What are your views? Have you had experiences of semantic search? What do you think is the future of Search? Do you use an alternate search engine? As always we’d love to hear your opinions leave a comment and share your thoughts!