16 of 17

Write now Article Tools

Page 3 of 3

require your computer sit idle for the whole day. Even with cleverer systems, with carefully synchronised perl and sql commands, with data which has been tagged and linked and all kinds of weird things have been done to it to aid swift matching - even these things end up with upper limits that close in on the programmer when the data exceeds a few 100,000 items. I know because I tried to build stuff from scratch myself, years ago, and I've seen these structures collapse from the inside!

Google didn't really do anything miraculous, in programming terms. They don't have a clever search query system which is better than all the ones I mentioned - no doubt they USE those methods. What they did which was different is they ran the whole process in advance - performed keyword matching on EVERY page they crawl and put it in some kind of library. When a person types keywords, they are being routed to the libraries Google has already got ready. A so-called search of billions of pages is really just a retrieval of 100s of pages of data with literally nothing more complex in the processing than that. The search script is NOT where the work is done, the crawler script is.

What we can hope for

Google is by no means the end of the story. Their method still doesn't really give us a perfect result, merely the best available. Other ideas for how to get an even better result WILL come into play, not all of them thought up by Google. One of the things on the cards is the 'semantic web' - and I think that when some bright spark somewhere has found the most ingenius way (for now) of handling the content worldwide with 'semantic' related software, seeking to enable a more 'semantically aware' form of search, we will be able to access the 900,000 results which Google can't actually let us see, after we've scoured all the pages of results they show us when they tell us there's a million. Where or when or how this semantic step will be taken is almost impossible to predict - all you can be sure of is that it won't come from any of the mainstream efforts to produce it. Google did not, nor does any 'sudden' change.

In practical terms we really need to be able to know what a page is 'about' rather than what the keywords are. More and more these days it is not enough to just match keywords to a page, not just because of how many ruthless advertising networks promote the development of sites purely for the purpose of directing people at products according to what keywords they seek, but also it just makes sense that the engine needs to establish, for example, which war related sites are pro-war and which are anti-war, or which sites have immaculate spelling and grammar and which are of a lower quality. We need to be able to guage what we are getting a lot better than we can do even with Google.

The natural result of development of this kind would be to provide a platform which would be invaluable to sales people and corporations but, like Google, I believe its primary port of call will be to serving the international community with information from the world of academia, research, music and current affairs. The reason for the allegiance of good information systems to the sectors of society which most deserve it is quite simple - it is from those sectors that the answers will come and the new solutions will come into being. Above all it is likely to happen via Linux operating systems, just like Google happened that way, because this environment affords great sophistication and power at zero cost. I estimate that by 2020 such a leap will have happened.

Learn more about this author, Two one three four six six.
Click here to send this author comments or questions.

• << previous page
• 1
• 2
• 3

Add your voice

Know something about Search engine evolution: Where they started and where they are today?
We want to hear your view. Write now!