16 of 17

Write now Article Tools

Page 2 of 3

overwhelming volume of data inside just one server, let alone across ten billion websites, is the source of the problem which is solved by the sorts of technique most visible in search engines. Since all decent servers are housed in Linux operating systems, it is no surprise that from the Linux commandline you can actually search for things yourself with a single command - 'grep'. And inside the perl language (one of the most flexible ways to handle a Linux server) pattern matching turns out to be a highly developed artform with all manner of different subtle ways to deal with it.

Though there are formalized and standardized ways of dealing with this stuff, what I find most enlightening in this issue is the basic structure of handling these acts of seeking and finding. For example from the commandline, grepping it enables you to specify a file or general location to look in, and then is totally free, enabling you to just type grep * "harry potter" and it'll root it out for you anywhere in your current folder. You could add /* to the * and indeed add many, and it would search deeper down the hierarchy. More cunningly you could write a perl script to produce a shell script which consisted of maybe a dozen commands and in one fell swoop you could search the whole machine for the term.

In perl, matching is more complex and therefore the process can be better controlled - it won't take two hours to do something which, with a little lateral thought, you could arrange to have done in seconds.

Naturally other languages and 'communication methods' in the broad world of domstic and business computers all have their equivalent - sql, common to Linux and to other server/system environments, has a basic facility for seeing if something equals something or is 'like' something (not actually hugely complex, a lot less useful than it sounds), and of course your various microsoft or cross-platform tongues, from asp to php, are all bound to have crude matching commands also.

In the end what made Google the King of the castle is quite simple. None of those matching systems, whether the sophisticated Linux enquiry-farms or the knuckle-dragging php-meets-sql blood-from-a-stone enquiry, can really face up to the alarmingly difficult task of scanning billions of objects in order to find one or more precise things in their content... raw sql would, for example, lead to a query taking between a quarter of an hour and a few hours - the equivalent search on Google for just one thing could

• << previous page
• 1
• 2
• 3
• next page >>

Add your voice

Know something about Search engine evolution: Where they started and where they are today?
We want to hear your view. Write now!