by Wendy Connick

Created on: June 17, 2008   Last Updated: October 13, 2010

Search engine optimization (SEO) is a hot topic these days for webmasters and content writers alike. SEO means designing your web site so as to get the best possible response from search engines. This doesn't just mean getting your site to show up in a lot of searches; they have to be the right searches, or your visitors will leave your site in disgust as soon as they realize that you don't have what they are looking for. The key to effective SEO is understanding how search engines work.

Search engines use a type of software program called a spider, or crawler, to visit web pages. Each web page is "crawled" and then indexed, meaning that it's added to the search engine's list of active web pages. When a spider visits a web page it will follow every available link to get every page on the site indexed.

Because of the sheer volume of material on the web, crawling each and every page is extremely resource-intense, even though spiders can crawl pages at an average rate of 100 pages a second. Different search engines use different indexing strategies to best manage their resources. For example, Lycos' spiders look at specific parts of a web page such as headings and links, and crawl every word in the first twenty lines of each page. Google's spiders skip over "a," "an," and "the" and other articles of speech, but index every other word on every page. AltaVista crawls every single word, including the basic articles of speech.

Once search engines have collected this data, they build an index: a list of all the words and URLs which the spiders have gathered from all over the Web. This index doesn't just contain a list of words, however; search engines use a number of tricks and techniques to ensure that the most relevant pages for your search appear at the top of the list, while the least relevant will be clustered at the bottom.

As an example, try firing up Google and enter a search for "jade elephant." Even this fairly obscure search phrase returns about 175.000 results as I'm writing this, and obviously not even the most dedicated jade elephant-lover is going to wade through hundreds of thousands of web pages.

That's why Google and other search engines try to present the best pages for your search at the beginning of the list. To accomplish this, search engines use "weighting" techniques to rank some pages above others for different search terms. The specific weighting techniques for each search engine are kept secret by their programmers. SEO experts observe searches and keyword results to deduce how the spiders rank pages, but as soon as a weighting technique becomes widely known, the search engine programmers will switch to a different technique. This helps keep unscrupulous webmasters from "tricking" the search engines into listing their sites inappropriately.

There are a few constants among the major search engines. Most search engines will give more weight to a word or phrase that's repeated several times in a specific page. They also tend to pay extra attention to words in important parts of the page, such as text inside a header or hyperlink. A page's meta tags are also evaluated closely, particularly the title tag.

Now that you've had a quick glimpse of how search engines work, you have a leg up on designing your own pages to be as attractive as possible for those little spiders!

Learn more about this author, Wendy Connick.
Click here to send this author comments or questions.