About Inktomi Slurp, Yahoo Slurp & Googlebot
Optimizing web pages for search engines such
as Inktomi and Google is a great start, but what good is all
your hard work if the those search engine spiders never stop
by for a visit? The two primary spiders to focus on are Inktomi's
"Slurp" and Google's "Googlebot".
is Inktomi's web-indexing robot that strolls the Internet
collecting information from individual web pages and compiles
them into a searchable index used by companies such as HotBot
and MSN for search results. Slurp is a standalone UNIX based
robot with a sole purpose defined as indexing / statistics
and adheres to the Robot
If you simply create a web page and host it
on a server you'll have to patiently wait for Slurp to stop
by for a visit and collect your information to be included
in Inktomi's search results. This means checking server logs
for specific ip addresses on a regular basis to see if and
when Slurp stopped by for a visit. You also have to wait longer
to see your optimized web pages show up in Inktomi results.
If you've signed up for our Inktomi
Paid Inclusion then your life just got easier.
Within 48 hours your site will be spidered by Slurp and your
guaranteed frequent return visits about every other day. So
you know your getting spidered but how do you know what pages
are being crawled?
Yahoo Slurp - NEW
Yahoo Slurp is Yahoo's search engine spider
(similar name to Inktomi's - completely different spider).
Similar to Googlebot, Yahoo Slurp follows every instance
of HREF links and dynamic links but not SRC links such as
image and frames. If you want to make sure your individual
pages get spidered, make sure your linking menus are in working
order and just to be safe create a site
map. Yahoo Slurp crawls your web pages to be indexed
in the Yahoo search database and in return listed in it's
Before you worry about getting each and every
one of your pages spidered, you need first and foremost to
get Yahoo Slurp to your homepage. If you want to speed up
this process you can submit your web pages to the Inktomi
network through Inktomi
Paid Inclusion. While Yahoo Slurp will find
your web page on it's own, the paid inclusion guarantees
Slurp visits and often at that, allowing you to quickly get
your pages index and hopefully listed well in the SERP's.
For more information on Yahoo Slurp visit Yahoo's
Slurp FAQ Page.
Crawler Listings - That's how!
SpyderTrax is a program created by Darrin Ward
that displays the frequency and magnitude of individual search
engine spider visits to individual web pages.
The link below is what we use to check how often
what search engine spiders are visiting what pages.
This easy to read representation gives instant
feedback on what individual spiders have stopped by. Forget
searching out the individual ip address of the spiders, SpyderTrax
simply tells who came by and when. As you can see by the hits
from Inktomi Slurp, we are averaging hits almost
Installation Offer - Only $100
SpyderTrax is a program which needs to be
installed on all web pages you wish to see spider hits from.
We offer installation of SpyderTrax on ALL of your
web pages for only $100. If your interested in
getting SpyderTrax installed click the order button below.
In additional to seeing spider hits from Inktomi and Google,
you also receive stats from Altavista, Microsoft, Fast Search,
Infoseek, DMOZ, Wisenut, Teoma and many more!
is Google Inc's search engine spider. Googlebot is a standalone
Linux based robot with a sole purpose of indexing web pages
and adheres to the Robot
Exclusion Standard. Unlike Inktomi which changes
results every few days, Googlebot crawls web pages over the
course of a few weeks and then has it's anticipated Google
The Google Dance is basically the main index
update of the Google search engine. When this takes place
(typically once a month) Google results will change depending
on which data serving the results are pulled from. The results
change sporadically at first and then settle down as the dance
ends usually lasting between 3-5 days. You can view our Googlebot
hits (courtesy of SpyderTrax) by clicking the link below.
Googlebot loves new content! New web pages
are spidered by Googlebot often, varying around every few
days to check and see if new content has been added. If
so, you can expect frequent Googlebot visits. If your web
pages rarely change and have no updates, Googlebot my place
you into it's monthly visit, right before the dance. Why
should Googlebot come for a visit if your web page has nothing
new to say?