1 of 5

Write now Article Tools

To be found on the Web, think like a spider

The World Wide Web, commonly referred to as the Web, resembles a gigantic and intricate invisible spider web that connects web sites together through hyperlinks. Much like how a spider's web is composed of many individual strands that link different parts together, the Web is linked in a complex series of relationships and connections. Finding any individual website that is not dominant in the public or consumer eye is impossible without help. However, the search engine technologies that permit us to search on our favorite topics use an arachnid-like system to index content. If you want your website to be found on the Web, think like a spider.

In real life, spiders not only create webs, but they use them to travel from one place to another. Each web is made up of individual links that connect its components. However, each part of a spider's web is not directly linked to every other part of the web each web strand normally connects to a handful of other strands. A spider may have to move across many strands in order to go from one point to another. It's also a much slower process to get from one point to another (i.e. from one corner of a barn to the other corner of the barn) unless there is a network of web strands that can be used to get there.

The creators of search engine technology came up with a practical means to catalog the millions of pages on the Web. They created software applications called "spiders" that explore the Web and make note of web sites for future cataloging. Put simply, spiders have a starting place from which they follow links (crawling in their quest to find web pages. Once they find a web site, they follow all links as best they can and move on to the next site.

However, if web pages aren't linked to other pages (like separate, unconnected webs), if the links to these web pages are inaccurate (broken web strands), or if the web pages in question don't exist anymore, then the spiders have a very tough job in cataloging those pages. This means that the search engines that they serve may not catalog these problematic links and pages for a long time, if ever. There's plenty of crawling to do, so if it's hard to do, it won't happen very quickly.

The lesson: make it easy for the spiders to crawl to your site. Make sure you build links to other web sites. Keep your links up to date and accurate. Even better, give the spiders an even better chance of cataloging your site by creating a sitemap file that lists all of your pages. If you want to make it easy for people to find your web site, make it easy for the spiders to get there. . If you want to be found on the Web, think like a spider.

Learn more about this author, Mark Dykeman.
Click here to send this author comments or questions.

Add your voice

Know something about To be found on the web, think like a spider?
We want to hear your view. Write now!