Robots.txt

A file that instructs search engine bots which pages to crawl and index.

Robots.txt is a text file placed at the root of a website (e.g., example.com/robots.txt) that provides instructions to web crawlers about which pages or sections of the site should not be crawled or indexed. It uses Allow and Disallow directives per user-agent. While compliance is voluntary for legitimate crawlers like Googlebot, it is respected by all major search engines and AI crawlers.