Important Search Engine Robot User-Agents
This portion includes a list of all the spider user-agents of the important search engines. The versions on this list will eventually go out of date, but the list will remain useful by helping to identify oddly named spiders (Ex. IA Archiver = Ask.com).
Common Robot Traps to Avoid
This box includes a list of the most common ways webmasters unintentionally stop spiders from crawling their sites.
Robots Meta Tag Syntax
This section includes documentation for the robots meta tag. This includes all of the available arguments as well as search engine compatibility.
Robots.txt Syntax
A example of a simple robots.txt. This illustrates how to block specific robots from both entire directories and specific files.
Sitemap Syntax
This section shows the standardized sitemap.xml syntax. It also lists the default places search engines look for sitemaps.
This portion includes a list of all the spider user-agents of the important search engines. The versions on this list will eventually go out of date, but the list will remain useful by helping to identify oddly named spiders (Ex. IA Archiver = Ask.com).
Common Robot Traps to Avoid
This box includes a list of the most common ways webmasters unintentionally stop spiders from crawling their sites.
Robots Meta Tag Syntax
This section includes documentation for the robots meta tag. This includes all of the available arguments as well as search engine compatibility.
Robots.txt Syntax
A example of a simple robots.txt. This illustrates how to block specific robots from both entire directories and specific files.
Sitemap Syntax
This section shows the standardized sitemap.xml syntax. It also lists the default places search engines look for sitemaps.
0 Comments