What sort of world do search engines reflect, and does it have room for more than one cultural viewpoint, asks Haydn Shaughnessy

There can be few activities we engage in so often (in America and Europe we do it about 8 billion times a month), fail at so frequently (only 50 per cent of the time are we successful), and yet still feel happy doing it (85 per cent of all Internet traffic goes through them). We are talking search engines and how we find things out.

Ten years ago the search engine, as a way of life, was a relative unknown. Today it is difficult to conceive of an area of a daily activity that is not facilitated in some way by an online search. If a sociologist were looking for a prime candidate for cultural innovation of the decade, a search engine would most likely be it.

"Search has been accepted as the commonest way of interacting with information," says Bjorn Olstadt, chief technical officer of Fast, a European leader in information search services, "the default way of understanding and navigating information".

And that's why the search engine is fast becoming a political football with French and German commercial and cultural interests, in particular, rising up the European agenda. Only five years ago the most efficient and largest search engine on the world wide web was a European one, Alltheweb.

Produced by a group of savvy Norwegians, but marketed from the US, Alltheweb used a fraction of the processing power that its ultimate nemesis Google did, and indexed far more of those world wide web pages. But in web terms Google eclipsed Alltheweb, Alta Vista, Lycos, Northern Lights, Ask Jeeves, and many, many more, all search engines that were in the race for global dominance a few short years ago.

When we think search, we now think of the simple elegant interface that asks if we feel lucky and which, lucky for its investors, now hogs 75 per cent of web searches.

For a number of reasons that's wrong, or so said the French President, Jacques Chirac, and the former German Chancellor Gerhard Schroeder back in the middle of last year.

Between them they stumped up more than €300 million to pay for the development of the next generation of search engine, based around European cultural needs. The two leaders decided Google had become too powerful in our everyday contact with information, a fear that found extra justification when the American search engine agreed to scan and index the libraries belonging to Harvard University, Stanford University, Oxford University, the University of Michigan and the New York Public Library, potentially giving an English language perspective on world history and culture a decisive online advantage over others.

The Google libraries initiative ran into a number of legal problems. However, Chirac had already responded with a plan to promote a similar project for French and European works, to create "a diversity of influence" on the internet.

Chirac's moody riposte has evolved into a mission to define what we Europeans lose by being served in our everyday information needs by an American search engine, and then to correct that. Known by the name QUAERO, it is a comprehensive plan to develop what we might be using to find information in five years' time. Alongside QUAERO, the European Commission has now launched a €50 million European programme of research into how we might in future go about finding out.

What might be the problem with Google?

Search engines serve our needs in just about every sector of the economy, governance and leisure: in sovereignty and security (all those images from CCTV cameras and image databases need trawling by a machine that recognises similarity), in e-Government (think Revenue Commissioners, the Health Service Executive, Ministries of this or that), health queries when we distrust the doctor, homework, cooking, shopping, travel, tracking our goods, and tracking the various components that make up our cars, bikes, houses, planes, music, relationships, films, television.

It is not just that the world wide web, which delivers much of this information, is a fantastic information resource. That, in a sense, is an uninteresting observation. The world of non-world wide web information happens to be growing faster. With an increasing volume of information coming from multimedia sources (music, films, podcasts, photographs) and with 10 billion text pages on Google alone, what matters is that finding out is possible at all, despite all the failures. Finding half of what we need is in fact a miracle. The issue is whether we find information of real relevance or we accept too readily what search engines deliver.

Has searching displaced some other activity? Turning to a neighbour and asking a question? Memorising the information we need to know? Visiting a shop and discoursing with a travel agent? Trusting a doctor or similar professional? Making judgments about people and their reliability when imparting information?

The answer to those questions could define how new search engines are built and how we will interact with machines in the near future.

And therein lies a possible clue to the fortunes of Google and its potential competitors.

Google's elegant screen presence has made it the dominant player in the search world. In terms of performance, according to Alltheweb's original developers, Fast, it never had the edge. "We were more efficient by a factor of five to 10 times," says Olstadt, whose company no longer provides web search services.

Google, through venture capital funding and then a public offering of its shares, has managed to garner the resources to become the Microsoft of its genre. But more than that, Google has kept it simple. Its proposition was that by taking into account the links users make between their websites, it allowed us all an implicit vote on which sites should top its search return rankings.

This is not quite what it seems, though.

There is now a substantial industry geared toward distorting Google search returns, according to Professor Tobias Scheffer at the Humbolt University of Berlin.

"There are whole server farms dedicated to the dissemination of biased or untruthful information, and the imitation of relevance rankings," says Scheffer, who cites "link spam" as one of the most perfidious. Link spam is created on link farms, rooms with potentially hundreds of servers, the machines that power the web, creating thousands of links between websites in order to get noticed by the major search engines.

The word is, according to Scheffer, that half of all web servers may now be involved in link spam. This, says Scheffer, "means we are measuring relevance [of information] in the presence of purposeful obfuscation".This alone, argue its critics, is enough to bias Google in ways we cannot understand but that most certainly puts commercial interests ahead of cultural, a distortion that is a product of Google's basic technology. This, rather than French political and cultural fears, might be the most important context for innovations that ultimately might challenge Google's position.

Google, and all the search engines left in its wake, misses the point of networked computing, according to experts such as Scheffer. Today the power lies with networks of people as much as networks of machines.

The answer to obfuscation, error and failure, in an ever-expanding universe of data, and possibly the key to future search engines, are the human networks contributing their own content to the general pool of wisdom out there.

Blogs, the online encyclopaedia Wikipedia, personal journals and the social networks that grow up around these are, some experts now argue, an ideal way of bringing the human touch back into the business of searching.

If you are searching for information, the chances are somebody else has already been there. Capturing their experience for the good of those that come next is one important goal of the next generation of search engine.

Put simply, if your passion is food, you're in Paris, and you have a search engine that connects you with the eating habits and restaurant preferences of French gourmets, and allows you to operate across the language barrier, you may never have to read another article on Paris in an airline magazine or the weekend travel section again.

Bringing people back into search and retrieval, and using computing power to interpret their recommendations and search patterns, might also help address the exclusion of non-English language cultures from the internet mainstream. And in a highly mobile world, relating your search requirements to where you are, at what time of day, and your role as father, mother, employee, or traveller, also adds relevance.

By analysing the behaviour of groups and by allowing the reputation of trusted sources of information to float above the scurrilous, future search engines will harness the traditional power of the meeting place, whether it be the pub or the mart. That's why Google has now embarked on buying companies with strong social networks at their base.

It seems that with all their faults, human systems are now being recognised for a power that is every bit as extraordinary as that of the computer. Vive L'Egalité, as the French might say.