AI Crawler Audit

Included

• Audit of robots.txt and Cloudflare settings per domain
• Baseline documentation before 15 September 2026
• Bot classification by effect, checked against operator documentation
• Reasoned decision per domain, in writing
• Follow-up check in Search Console and server logs

How we work

1. Inventory: every domain behind Cloudflare, plus its current robots.txt verbatim
2. Secure the baseline: screenshot of bot settings, crawl statistics export, robots.txt archived — each with a date
3. Classification: which bot sits in which effect class, and what a disallow there actually costs
4. Check where the content worth protecting actually lives — a block on one domain secures nothing if the same content sits open elsewhere
5. Decision per domain, reasoned in writing, then implementation in robots.txt and Cloudflare
6. Fetch the served file as evidence, then a follow-up check after two weeks

What you concretely get

• For every domain it is documented what it permits AI providers to do — and why
• The state before the deadline is provable, not remembered
• No training block that secures nothing because the same content sits open elsewhere
• Not for you: if you are not behind Cloudflare and allow model training anyway, there is nothing to do and you do not need this audit

Engagement model

One to two days for up to ten domains, plus a follow-up check after two weeks. No retainer.