AI search
AI crawlers and robots.txt: what to allow
Training crawlers, search crawlers and user-triggered fetchers do different jobs. Here is how to allow AI search bots in robots.txt and block training.
Three kinds of AI bots
AI companies send different bots for different jobs, and robots.txt lets you treat each one separately. The useful distinction is what a visit is for:
- Training crawlers
- Collect pages that may be used to train AI models. Blocking them is a training decision, and for most vendors it is separate from search visibility.
- Search crawlers
- Build the index an AI search engine retrieves from. Blocking them can keep you out of that engine's answers.
- User-triggered fetchers
- Visit a page because a person asked the assistant a question or asked it to open a link. They act on one request rather than crawling.
Documented AI user agents
- GPTBot
- OpenAI. Crawls content that may be used to train its models.
- OAI-SearchBot
- OpenAI. Surfaces and links sites in ChatGPT search results.
- ChatGPT-User
- OpenAI. Visits pages for user actions in ChatGPT and custom GPTs.
- PerplexityBot
- Perplexity. Indexes sites to surface and link them in Perplexity answers. Perplexity says it is not used to train foundation models.
- Perplexity-User
- Perplexity. Fetches pages when a user's question needs them.
- ClaudeBot
- Anthropic. Collects web content that may be used to train its models.
- Claude-SearchBot
- Anthropic. Indexes content to improve search results in Claude.
- Claude-User
- Anthropic. Fetches pages when a Claude user asks a question.
- Google-Extended
- Google. A robots.txt token, not a separate crawler. It controls whether content Google crawls may be used to train Gemini models and ground Gemini answers. It does not affect Google Search or AI Overviews.
- Applebot-Extended
- Apple. A token, not a crawler, that controls whether content Applebot collects may be used to train Apple's foundation models.
- CCBot
- Common Crawl. Builds an open web archive that many organizations, including AI developers, use as training data.
- Bingbot
- Microsoft. Crawls for Bing's index, which also supplies web results to Microsoft Copilot.
User-triggered fetchers follow different rules
Robots.txt was designed for automated crawlers, and the standard (RFC 9309) treats it as a request, not access control. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the visit, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. To keep a page private, use authentication. Vendors revise these policies, so check their documentation.
Allowing search while blocking training
A common setup allows the search crawlers, so you can be cited, and blocks a training crawler you do not want. A crawler that finds a group naming it follows only that group and ignores the catch-all group, so repeat any shared rules in each group.
# AI search crawlers
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Disallow: /admin/
# Model training
User-agent: GPTBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xmlRobots.txt governs future crawling. It does not remove content a company has already collected.
Check the firewall and CDN too
CDN bot protection, web application firewalls, hosting security plugins and rate limits can block or challenge AI crawlers before they read robots.txt, and some CDNs offer a single switch that blocks AI bots across a site. This is a common silent mistake: robots.txt says allow, but the crawler gets a 403 or a challenge page and never sees your content.
- Search your server or CDN logs for each user agent and check the response codes it receives.
- Review bot management and firewall rules for AI or crawler categories set to block or challenge.
- Verify suspicious traffic against the IP ranges vendors publish, since anyone can fake a user agent.
Put it to work
- AI crawler access and llms.txtAI engines can only cite pages they are allowed to read. Here is what Nalza checks and fixes.
- Free auditCheck your homepage in about a minute and see a preliminary score with your top issues.
More in AI search
- What is generative engine optimization (GEO)?GEO is the practice of making your site easy for AI answer engines to find, quote and cite. Here is how it relates to SEO and where to start.
- How do AI answer engines choose their sources?AI answer engines retrieve pages from a search index or a live fetch, then write an answer from them. Here is what that means for getting cited.
- What is Share of Answer in AI search?Share of Answer measures how often AI engines cite or mention you across a fixed set of prompts. Here is how to define it, sample it and compare it.
- How do Google AI Overviews work?Google AI Overviews summarize answers from Google's index and link to supporting pages. Here is what makes a page eligible and how to control it.
- What is llms.txt and do you need one?llms.txt is a proposed Markdown file that gives language models a short map of your site. Here is how it is structured and what it can and cannot do.
- How to write content AI engines can citeAI engines quote passages that answer a question clearly and stand on their own. Here is how to structure, source and maintain content they can cite.