Thu Jan 10 18:02:46 PST 2019

The robots
txt file (a product of the Robots Exclusion Protocol)
www
google
com/robots
txt)
The file serves as an access guide for
automated visitors (web robots)
By using robots
txt, webmasters can
indicate which areas of a site they would like to disallow bots from
crawling as well as indicate the locations of sitemaps files (discussed
below) and crawl-delay parameters
You can read more details about
this at the robots
txt Knowledge Center page

The following commands are available:
Disallow
Prevents compliant robots from accessing specific pages or folders

Sitemap
Crawl Delay
Indicates the speed (in milliseconds) at which a robot can crawl a
server

An Example of Robots
txt
#Robots
txt www
example
com/robots
txt
User-agent: *
Disallow:
User-agent: spambot
disallow: /
sitemap:www
example
com/sitemap
xml
Warning: It is very important to realize that not all web
robots follow robots
txt
People with bad intentions (e
g
, e-
protocol and in extreme cases can use it to identify the
location of private information
For this reason, it is
recommended that the location of administration sections
and other private sections of publicly accessible websites
not be included in the robots
txt
Instead, these pages can
utilize the meta robots tag (discussed next) to keep the
major search engines from indexing their high risk
content