Robots.txt

What robots.txt does, how Crawl-delay works, and exactly what this crawler honors when it audits your site.

What robots.txt is

robots.txt is a plain-text file at a domain's root (e.g. example.com/robots.txt) that tells crawlers which paths they may fetch. It uses User-agent / Allow / Disallow rules, can declare a Crawl-delay (minimum seconds between requests), and conventionally points to the site's sitemap location. Compliance is voluntary — it's a convention, not access control — but reputable crawlers honor it.

What this crawler honors

• Sitemap discovery — every audit starts by reading robots.txt and following its sitemap references, which is how the crawler finds sitemap indexes on sites that don't use the default /sitemap.xml path.
• Crawl-delay — if the file declares a crawl delay, the audit spaces its requests accordingly, on top of its own randomized jitter between requests.
• Disallow rules — the audit only fetches publicly accessible pages and never attempts logins or form submissions; paths the site asks crawlers to avoid are left alone.

Polite crawling, concretely

Beyond robots.txt: rotating realistic browser user-agent strings (not a bot-identifying UA), randomized delays between requests so traffic doesn't arrive in mechanical bursts, sequential page rendering in browser mode, and a single egress with no IP rotation. The goal is to behave like a well-mannered visitor, not a scraper swarm.

What robots.txt can't do

It can't keep a page out of search results (that's what noindex is for), it can't hide URLs from anyone who guesses them, and a Disallow doesn't prevent the audit from listing a URL that appears in the public sitemap — it only governs fetching. If you need a page truly private, it needs authentication, not a robots rule.