robots.txt Generator and AI Crawler Audit

How to use

1. Build tab: one rule per line (allow /admin/ or disallow /admin/), comma-separate multiple User-agents, one URL per line for Sitemap.
2. Matching is longest-first, not last-write-wins: when Allow and Disallow both match, the longer path wins, and Allow wins a tie.
3. Disallow is a prefix match, so /admin also blocks /administrator — the classic collateral damage, and nothing will ever warn you. The audit tab calls these out.
4. Once a specific agent group matches, the * group no longer applies, even if that group has no rules at all. An empty group is therefore a silent allow, and the audit warns about it.
5. The AI crawlers tab computes the real policy for 25 crawlers whose token is stated in vendor docs: all sample paths blocked reads as blocked, none as allowed, anything else as partly blocked.
6. The block-training-crawlers preset emits two groups: the * group keeps your own admin/api rules, and the training crawlers get their own group with Disallow: /. Collapsing them into one group would block * site-wide too.
7. robots.txt is a convention, not a permission. Compliant crawlers read it and harvesting crawlers ignore it — real access control needs auth or a WAF.
8. llms.txt is a proposed format from 2024, not a standard, and no vendor has promised to read it.

Related tools