FREE CRAWLERS - Free Downloads

Manage and track visits by robots, spiders and crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Easy to useBy placing a Sitemap-formatted file on your webserver, you enable the Google crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.ETagFix is an ISAPI plugin which integrates directly into IIS and other web servers to alter the etag that IIS produces so that web crawlers will not check your web site so often.