All About Inktomi Slurp, Yahoo Slurp & Googlebot

Optimizing web pages for search engines such as Inktomi and Google is a great start, but what good is all your hard work if the those search engine spiders never stop by for a visit? The two primary spiders to focus on are Inktomi's "Slurp" and Google's "Googlebot".

Slurp is Inktomi's web-indexing robot that strolls the Internet collecting information from individual web pages and compiles them into a searchable index used by companies such as HotBot and MSN for search results. Slurp is a standalone UNIX based robot with a sole purpose defined as indexing / statistics and adheres to the Robot Exclusion Standard.

If you simply create a web page and host it on a server you'll have to patiently wait for Slurp to stop by for a visit and collect your information to be included in Inktomi's search results. This means checking server logs for specific ip addresses on a regular basis to see if and when Slurp stopped by for a visit. You also have to wait longer to see your optimized web pages show up in Inktomi results.

If you've signed up for our Inktomi Paid Inclusion then your life just got easier. Within 48 hours your site will be spidered by Slurp and your guaranteed frequent return visits about every other day. So you know your getting spidered but how do you know what pages are being crawled?

Yahoo Slurp is Yahoo's search engine spider (similar name to Inktomi's - completely different spider). Similar to Googlebot, Yahoo Slurp follows every instance of HREF links and dynamic links but not SRC links such as image and frames. If you want to make sure your individual pages get spidered, make sure your linking menus are in working order and just to be safe create a site map. Yahoo Slurp crawls your web pages to be indexed in the Yahoo search database and in return listed in it's SERP's.

Before you worry about getting each and every one of your pages spidered, you need first and foremost to get Yahoo Slurp to your homepage. If you want to speed up this process you can submit your web pages to the Inktomi network through Inktomi Paid Inclusion. While Yahoo Slurp will find your web page on it's own, the paid inclusion guarantees Slurp visits and often at that, allowing you to quickly get your pages index and hopefully listed well in the SERP's.

For more information on Yahoo Slurp visit Yahoo's Slurp FAQ Page.

SpyderTrax is a program created by Darrin Ward that displays the frequency and magnitude of individual search engine spider visits to individual web pages.

The link below is what we use to check how often what search engine spiders are visiting what pages.

http://www.web-page-optimization.com/cgi-bin/spyderview.cgi

This easy to read representation gives instant feedback on what individual spiders have stopped by. Forget searching out the individual ip address of the spiders, SpyderTrax simply tells who came by and when. As you can see by the hits from Inktomi Slurp, we are averaging hits almost every day.

Special SpyderTrax Installation Offer - Only $100

SpyderTrax is a program which needs to be installed on all web pages you wish to see spider hits from. We offer installation of SpyderTrax on ALL of your web pages for only $100. If your interested in getting SpyderTrax installed click the order button below.

In additional to seeing spider hits from Inktomi and Google, you also receive stats from Altavista, Microsoft, Fast Search, Infoseek, DMOZ, Wisenut, Teoma and many more!

Googlebot is Google Inc's search engine spider. Googlebot is a standalone Linux based robot with a sole purpose of indexing web pages and adheres to the Robot Exclusion Standard. Unlike Inktomi which changes results every few days, Googlebot crawls web pages over the course of a few weeks and then has it's anticipated Google Dance.

The Google Dance is basically the main index update of the Google search engine. When this takes place (typically once a month) Google results will change depending on which data serving the results are pulled from. The results change sporadically at first and then settle down as the dance ends usually lasting between 3-5 days. You can view our Googlebot hits (courtesy of SpyderTrax) by clicking the link below.

http://www.web-page-optimization.com/cgi-bin/spyderview.cgi?date=all&listbot=Google

Googlebot loves new content! New web pages are spidered by Googlebot often, varying around every few days to check and see if new content has been added. If so, you can expect frequent Googlebot visits. If your web pages rarely change and have no updates, Googlebot my place you into it's monthly visit, right before the dance. Why should Googlebot come for a visit if your web page has nothing new to say?

	Add URL to Yahoo Pay Per Click Software	SEO Web Marketing Forum Website Promotion Products Froogle Product Search