Create a purpose table before copying a user-agent list
One provider can publish several identifiers with different purposes. OpenAI distinguishes OAI-SearchBot for search visibility from GPTBot for potential model training. Anthropic separately documents Claude-SearchBot, ClaudeBot and the user-triggered Claude-User. Record the purpose, policy decision, reason, owner and review date for each token so a training preference is never mistaken for a search-discovery setting.
Some robots tokens control use but do not crawl independently
Google-Extended and Applebot-Extended are important examples. They express whether content already fetched by the main crawler may be used for specified generative or training purposes; they are not independent HTTP crawlers that can be verified with a simulated request. An audit must distinguish a rule found in robots.txt from an actual network response, otherwise a purpose token may be reported as a successful or failed fetch when no such fetch exists.
Robots policy, HTTP access and rendered content are separate tests
robots.txt is a policy followed by compliant crawlers. A 403 or challenge page means the request was refused by an edge or origin control. A 200 still does not prove the crawler received the intended page because regional routing, authentication or client-side rendering may produce a different version. For representative user agents, fetch the homepage, service pages, robots.txt, sitemap.xml and llms.txt, then record status, final URL, content type and canonical.
Regional routing needs explicit crawler testing
IP- or browser-based routing can send a crawler to another country site and create mismatched language, canonical and hreflang signals. Exemption lists also age as providers introduce identifiers. Test representative search, user-fetch and training agents on a schedule. When a new identifier appears, confirm its official purpose before deciding whether it should bypass routing.
Edge platforms can expose several independent controls
Managed robots content, AI crawler controls, general bot management, WAF rules, rate limits and cache can all affect the result at the same time. Switching off one control does not disable the others. After a policy change, clear the relevant robots or page cache, repeat the test externally with each user agent and compare it with a direct origin test to locate the responsible layer.
Redirect only to genuinely equivalent content
When an old article has a true replacement, redirect it to the closest equivalent page. Sending many unrelated URLs to a category or homepage may be treated as a soft 404. Where no replacement exists, decide between 404 and 410 after considering traffic, links and compliance obligations. Keep a migration table of old URL, target, status and reason, then validate redirect chains, canonicals and sitemap membership after release.
Turn policy into a repeatable acceptance sheet
The minimum sheet should record the origin robots content and its single sitemap reference, whether an edge service injects managed content, final status and URL for each tested agent, language and canonical signals on representative pages, valid structured data and a sitemap containing only canonical 200 URLs. Rerun the same checks after changes to routing, CDN, robots or application paths.
Practical next step
- Free Website SEO & GEO Audit: Review public structure, canonical signals, robots policy and AI-readable content.
- Free 404 & Redirect Checker: Find broken URLs, redirect chains and legacy pages that need individual treatment.