Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
网站信息
网站名称：The Web Robots Pages
网站域名：www.robotstxt.org
免责声明：本站内容仅用于学习参考，信息和图片素材来源于互联网，如内容侵权与违规，请联系我们进行删除，我们将在三个工作日内处理。联系邮箱：chuangshanghai#qq.com（把#换成@）