Search Engine indexing software is nicknamed “Spiders” to reflect how they crawl through a web site's pages. Sometimes SEO specialists also refer to the software as "bots" - short for robots.

Spiders record several aspects of a page, including the actual text. In the record that is created and indexed, the Spiders identify the frequency of particular words on a page and this becomes part of a complicated algorithm that calculates the value. If one page contained the word “cancer” 4 times and another page contained the word “cancer” 12 times, plus they had the word “cancer” in the Meta Tags and maybe also the Title of the page, the second page would rank higher. The algorithm also calculates and ranks how words are connected to other words, such as car parts, car finance, car warranty for the web page of an auto dealership.

 

Allison Wallis in MRK 410 June 2005 found a good definition of Spiders from http://whatis.techtarget.com

.
 

" A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated."

"Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read."

In order to allow spiders to effectively crawl your site, it helps to have easy navigation that will allow all pages to be linked to all other pages through some simple menu bar - then the number of pages you have "indexed" can be high and this wil help the over all rankingof your site.

Googlebot is Google's web-crawling robot. Google freely describes basic information about this program and what it does at  www.google.com/bot.html

There may be some pages on a website that you do not want to show up in Google, for example pages that are password protected or other content with difficult to find links. To keep the Googlebot from following links on your pages to other pages or documents, you should add a meta tag into the header tags of your page. The code looks like this:

<META NAME="Googlebot" CONTENT="nofollow">