How AI Chatbots Choose Sources to Cite: 4 Engines Compared

AI chatbots choose their sources in two documented steps: a vendor crawler has to be allowed to fetch the page, and a retrieval layer then judges whether that page answers the exact question a person asked. Four things decide whether a page survives both steps.

• Allowing the search crawler, not the training crawler, is what keeps a page eligible to be cited.
• User-triggered fetchers follow different rules from crawlers, and OpenAI and Perplexity both say robots.txt may not stop theirs.
• Google names no AI-only crawler for AI Overviews or AI Mode, and ties eligibility to the ordinary Google Search index instead.
• Robots.txt edits are not instant: OpenAI and Perplexity both publish a figure of roughly 24 hours.

OpenAI, Anthropic, Google and Perplexity all publish precise, checkable rules for the first step, naming which of their agents does which job and what blocking each one costs. None of the four publishes the criteria it uses for the second step. That asymmetry is the practical answer to how AI chatbots choose sources: access is an engineering problem you can control completely, and quotability is a writing problem you can only influence.

How do AI chatbots choose which sources to cite?

AI chatbots choose sources through an access layer and then a retrieval layer. The access layer is robots.txt plus the vendor's own crawler: when a vendor's search crawler was never allowed to fetch a page, that page is not in the candidate pool at all, however well it is written. The retrieval layer then runs one or more searches for the question and selects the pages it judges able to support an answer. Google describes its own version of this directly, telling site owners that "the best practices for SEO remain relevant for AI features in Google Search" (Google Search Central, AI features and your website), and it sets its own eligibility bar for supporting links, which I have broken down separately in my post on which pages on a service business site get cited by AI. When I audit a site for AI visibility, the access layer is always the first thing I open, because no amount of rewriting repairs a crawler that was disallowed when the site launched. If nobody has reviewed your robots.txt since launch, review it before you touch a single paragraph of copy.

• Step one, access: robots.txt, the vendor's named crawler, and whether the page returns a crawlable response.
• Step one for Google specifically: the page also has to clear Google's own eligibility bar for supporting links.
• Step two, retrieval: the per-question judgement about which candidate pages support the answer.

Which crawler decides whether an AI chatbot can cite your site?

OpenAI, Anthropic and Perplexity each run a named crawler whose documented job is search, and that crawler is the one that decides whether its engine can cite your site. OpenAI writes that "OAI-SearchBot is for search" and that it "is used to surface websites in search results in ChatGPT's search features" (OpenAI, Overview of OpenAI crawlers). Anthropic writes that "Claude-SearchBot navigates the web to improve search result quality for users" (Anthropic, About Claude's web crawler). Perplexity writes that PerplexityBot is "designed to surface and link websites in search results on Perplexity" (Perplexity, Perplexity crawlers). Google names no equivalent AI-only crawler on its AI features page and instead ties eligibility to the ordinary Google Search index. The rule I apply on client projects is to allow all three named search crawlers explicitly rather than relying on a wildcard block that nobody has read in years.

AI assistant	What feeds its search answers	Fetcher used when a person asks it to read a page	What the doc says about robots.txt and that fetcher	Documented consequence of disallowing the search crawler
ChatGPT (OpenAI)	OAI-SearchBot	ChatGPT-User	"Because these actions are initiated by a user, robots.txt rules may not apply."	Sites that opt out "will not be shown in ChatGPT search answers, though can still appear as navigational links."
Claude (Anthropic)	Claude-SearchBot	Claude-User	Stated for the bots as a group: they "respect 'do not crawl' signals by honoring industry standard directives in robots.txt."	Disabling Claude-SearchBot "prevents our system from indexing your content for search optimization", "which may reduce your site's visibility and accuracy in user search results."
Google AI Overviews and AI Mode	The ordinary Google Search index, with no AI-only crawler named	Not covered on Google's AI features page, which scopes itself to AI Overviews and AI Mode	Google points to noindex, nosnippet, data-nosnippet and max-snippet as the controls for what Search shows	Google-Extended is offered to limit "AI training and grounding in some of Google's other systems", not Search itself
Perplexity	PerplexityBot	Perplexity-User	"Since a user requested the fetch, this fetcher generally ignores robots.txt rules."	PerplexityBot is the crawler "designed to surface and link websites in search results on Perplexity"

Does blocking a training crawler stop a chatbot from citing you?

Blocking a training crawler does not stop a chatbot citing you, because the vendors run separate crawlers for training and for search and treat the two permissions independently. OpenAI states that "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" and, separately, that "Each setting is independent of the others" (OpenAI, Overview of OpenAI crawlers). Anthropic draws the same split, saying ClaudeBot is the agent whose content "could contribute to model training" while Claude-SearchBot exists to improve search quality (Anthropic, About Claude's web crawler). Perplexity is blunter still, stating that PerplexityBot "is not used to crawl content for AI foundation models". What I see agencies get wrong here is pasting a generic "block the AI bots" snippet into a client's robots.txt, which quietly removes the client from the citation pool as well. If the client wants reach without contributing training data, disallow the training crawlers by name and allow the search crawlers by name.

• Training crawlers named in the vendor docs: GPTBot and ClaudeBot.
• Search crawlers named in the vendor docs: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
• OpenAI's crawler page names OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot.
• Anthropic's crawler article names Claude-SearchBot, ClaudeBot and Claude-User.

Do robots.txt rules still apply when a user asks a chatbot to read a page?

Robots.txt does not reliably apply to a fetch that a person triggered, and OpenAI, Perplexity and Anthropic word this differently enough to matter. OpenAI says of ChatGPT-User that "Because these actions are initiated by a user, robots.txt rules may not apply", and adds that ChatGPT-User "is not used to determine whether content may appear in Search" (OpenAI, Overview of OpenAI crawlers). Perplexity says of Perplexity-User that "Since a user requested the fetch, this fetcher generally ignores robots.txt rules" (Perplexity, Perplexity crawlers). Anthropic takes the opposite posture in print, stating that "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and documenting that disabling Claude-User "prevents our system from retrieving your content in response to a user query" (Anthropic, About Claude's web crawler). On the projects I lead, the practical consequence is that robots.txt is treated as a citation control, never as a privacy control. If a page must not be read by anyone, put it behind authentication rather than behind a Disallow line.

• Robots.txt reliably governs the scheduled crawlers, which is where citation eligibility is won or lost.
• Robots.txt may not govern a fetch a person triggered inside ChatGPT or Perplexity.
• Authentication, not robots.txt, is the control for content that must stay unread.

What is query fan-out, and why does it change which pages get picked?

Query fan-out is Google's own term for what its AI features do while composing an answer, described as "issuing multiple related searches across subtopics and data sources", with the note that "while responses are being generated, our advanced models identify more supporting web pages" (Google Search Central, AI features and your website). Query fan-out matters because the query your page is finally matched against is often not the query the person typed, so a page can be pulled in for a narrow subtopic it was never built to target. Google also limits when the feature appears at all, stating that "AI Overviews are only shown when our systems determine that it is additive to classic Search". In the content work I run for agencies, the practical response is to give each specific sub-question its own heading and its own self-contained answer, rather than burying six answers inside one long argument. If a section cannot be lifted out and still make sense, rewrite it until it can.

• One narrow question per heading, answered in the paragraph directly beneath it.
• Entities named in full inside each section, so a lifted passage still identifies its subject.
• No reliance on a special file: Google states "You don't need to create new machine readable files, AI text files, or markup to appear in these features". The separate case for structured data on a business site is covered in my write-up on Webflow schema markup.

What do OpenAI, Anthropic, Google and Perplexity say about how a page gets chosen?

None of OpenAI, Anthropic, Google or Perplexity publishes the criteria it uses to prefer one eligible page over another. Google publishes the most of the four about the behaviour around that choice: its AI features page describes query fan-out, states that "AI Overviews are only shown when our systems determine that it is additive to classic Search", and tells site owners that "the best practices for SEO remain relevant for AI features in Google Search" (Google Search Central, AI features and your website). On the crawler pages OpenAI, Anthropic and Perplexity publish, each vendor describes what its search agent is for and what blocking that agent costs, and none of the three sets out a rule for ranking one candidate page above another. The conclusion I work from on client projects is that eligibility is an engineering problem and quotability is a writing problem, and that hunting for an undocumented AI ranking factor is wasted budget. If a vendor has not published its selection rule, assume ordinary search quality signals are doing the work and write the page for a reader.

• Documented by Google: query fan-out, the additive-only condition for AI Overviews, and that SEO best practices still apply.
• Documented by OpenAI, Anthropic and Perplexity on their crawler pages: agent purpose and the cost of blocking each agent.
• Published by none of the four: the criteria used to pick one eligible page over another.

Where do sites lose AI citations by accident?

Sites lose AI citations by accident in three named ways, and all three are robots.txt mistakes rather than content mistakes. Every inherited site I audit gets checked for these before anything else, because each one is invisible in analytics and cheap to fix once found.

• The blanket AI-bot block. A generic "block the AI bots" snippet disallows the search agents alongside the training agents. Cost, in OpenAI's own words: a site that opts out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI, Overview of OpenAI crawlers).
• Robots.txt used as a privacy control. A Disallow line is a request, not a lock. Cost: the page stays reachable by a person-triggered fetch, because OpenAI says robots.txt rules "may not apply" to ChatGPT-User and Perplexity says Perplexity-User "generally ignores robots.txt rules" (Perplexity, Perplexity crawlers), so content you believed was shielded can still be quoted.
• The non-merging named group. Robots.txt groups do not merge: a crawler obeys only its most specific matching group. Google's robots.txt specification states that "User agent specific groups and global groups (*) are not combined". Cost: adding a bare "User-agent: OAI-SearchBot" group with no Disallow lines silently voids every Disallow in the wildcard group for that one agent, so staging and utility pages you thought were blocked become fetchable (Google, robots.txt specification).

How long does a robots.txt change take to show up in chatbot answers?

A robots.txt change takes about a day to land, and OpenAI and Perplexity both publish a figure of up to 24 hours. OpenAI states that "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results (OpenAI, Overview of OpenAI crawlers). Perplexity states that "Each setting works independently, and it may take up to 24 hours for our systems to reflect changes" (Perplexity, Perplexity crawlers). Google publishes no propagation figure for its AI features and gives a procedure instead, being a check of the markup with the URL Inspection tool plus time for recrawling. Anthropic publishes no propagation figure at all on its crawler article. Changing a client's robots.txt is a start-of-week job in my workflow, re-checked a few days later rather than the same afternoon. If an engine still shows the old state after that wait, treat it as a crawl or indexing problem rather than a robots.txt problem.

• OpenAI: roughly 24 hours for its search systems to adjust after a robots.txt update.
• Perplexity: up to 24 hours for its systems to reflect a change.
• Google: no propagation figure for AI features, and a procedure instead of URL Inspection plus time for recrawling.
• Anthropic: no propagation figure published, though it does document the non-standard Crawl-delay extension, which spaces out requests and is a different control from a propagation lag.

Which robots.txt and indexing checks should you run before publishing?

The pre-publish check for AI citation eligibility has four items and starts with robots.txt, not with the copy. Every site I take over gets the same pass, and on inherited sites the failure is nearly always a robots.txt line that predates the current AI landscape rather than anything about the content. The groundwork for the whole discipline sits in my guide to answer engine optimization. If any item below fails, fix it before writing anything new, because a blocked page cannot be rescued by better prose.

1. Open robots.txt and confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot are not disallowed.
2. Check every named user-agent group for its own Disallow lines, since a named group with no Disallow line voids the wildcard group's blocks for that agent.
3. Confirm the pages you care about are indexed in Google with the URL Inspection tool, and that no leftover nosnippet or max-snippet directive is suppressing their previews.
4. Decide the training question deliberately: disallow GPTBot and ClaudeBot by name if you want to stay out of training sets, and leave the search agents allowed.

A robots.txt that reflects all four decisions looks like the block below, and no vendor publishes a consolidated cross-engine version of it. Copy it, swap the Disallow paths for your own, and keep the repeated Disallow lines, because a named group without them silently opens everything the wildcard group blocks.

# Search agents: allowed, because these decide AI citation eligibility.
# Groups do not merge, so every wildcard Disallow is repeated by name.
User-agent: OAI-SearchBot
Disallow: /utility-pages/

User-agent: Claude-SearchBot
Disallow: /utility-pages/

User-agent: PerplexityBot
Disallow: /utility-pages/

# Training and other-systems grounding: blocked, without touching citations.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /utility-pages/

Sitemap: https://www.example.com/sitemap.xml

One file worth keeping separate from all of the above is llms.txt, which is an emerging convention rather than a citation control, and I covered what it does and does not do on Webflow in my post on llms.txt on Webflow sites.

Want someone to check which AI crawlers your site actually allows?

A 15 minute call is usually enough to go through your robots.txt line by line, name which of ChatGPT, Claude, Google and Perplexity can currently reach your pages, and tell you which single change would open the most ground. You get the finding on the call itself, not in a proposal afterwards. Book a 15 minute slot with me and bring the domain.