AI SEARCH CRAWLER
REFERENCE.

VERIFYING THAT A REQUEST IS GENUINE

Published endpoints · reverse DNS · the layer in front of the origin

A user-agent string is a claim, not an identity. Anyone can send one. What a provider publishes to settle the question is an IP list, a reverse-DNS pattern or a tool — and every endpoint below answered on the date in its row.

The table scrolls sideways.

Two things the documentation adds

Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. Anthropic · DOCUMENTED · 2026-09-27

The IP list is published for recognising Anthropic’s crawlers. Anthropic states that blocking those addresses is not a reliable way to opt out — robots.txt is the documented control.

If you’re using a Web Application Firewall (WAF) to protect your site, you may need to explicitly whitelist Perplexity’s bots to ensure they can access your content. Perplexity · DOCUMENTED · 2026-09-27

The check may have to happen in front of the origin. A robots.txt that permits a crawler does not help if a firewall or bot-protection layer answers that crawler with a challenge — a failure that is invisible in robots.txt and invisible in analytics. Perplexity’s documentation says so in as many words and lists firewall settings for it.

Where no endpoint is published

The CDN layer: a decision about training that reaches search

The layer in front of the origin can make its own decision about AI crawlers. Cloudflare, which serves this site, sorts them into Search, Agent and Training and documents two changes for 15 September 2026:

On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy “Block AI bots” option. Cloudflare · DOCUMENTED · 2026-09-27 · Cloudflare · AI bot policies (block AI bots)

Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service). Cloudflare · blog post of 1 July 2026 · DOCUMENTED · 2026-09-27 · Cloudflare Blog · new AI traffic options

Our reading: on a Cloudflare zone, a setting made about training reaches the crawlers behind Google, Apple and Bing search — and neither robots.txt nor the provider tables on this page would show it. INFERRED

The documentation was written before the date and says what Cloudflare “will” do. Whether the change took effect as described cannot be read from documentation — that would be OBSERVED, and it needs a zone’s own traffic. Other CDNs and bot-protection services have switches of their own; this page documents the one this site runs on. What we run on it is disclosed further down, next to our robots.txt.

THE THREE JOBS A CRAWLER DOES

Search · user retrieval · training

The tables on this page are sorted by what an agent does, because that is what the controls attach to. The same provider often runs one of each. This three-way split is our framing — INFERRED from the documented purposes, not a taxonomy any provider publishes — and the rows below say, per provider, where the documentation actually separates the three and where it does not.

The table scrolls sideways.

SEARCH BOTS

The agent to allow if you want to appear in that provider’s answers

The table scrolls sideways.

OpenAI also documents a fourth agent, OAI-AdsBot, which belongs to none of the three classes: OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. — DOCUMENTED · 2026-09-27 · OpenAI · crawler documentation.

Two entries that are absences

USER-RETRIEVAL AGENTS

One page, because a person asked — and robots.txt may not apply

These agents fetch a page because someone asked a question that needs it. Of the providers on this page that document one, five state a user-initiated exception to robots.txt — OpenAI, Perplexity, Google, Meta and Amazon, the last two in the six-provider table further down. Two document such an agent without stating an exception: Anthropic, which states blanket compliance, and Mistral. Whether that difference shows in practice is not something a table of documentation can answer — that would be OBSERVED, and it would need server logs. It is recorded as an asymmetry in the documentation, nothing more.

The table scrolls sideways.

• Not a permission. “May not apply” describes the provider’s behaviour. It does not make the fetch authorised, and it does not affect any other legal or contractual control you have.
• Not a search route. OpenAI: ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl. Allowing ChatGPT-User does not get a page into ChatGPT search.

TRAINING BOTS

A data-licensing decision — separate from search where the provider documents it as separate

Blocking these is a decision about your content. With the three providers below, the documentation separates that decision from search visibility — twice in the provider’s own words, once as our inference — and the column says which is which.

The table scrolls sideways.

The most misread control on this page

Google-Extended does not govern AI Overviews or AI Mode. Those are Search, and Search is governed by Googlebot. Google-Extended governs Gemini model training and grounding in Gemini Apps and Vertex AI, and Google states that it does not affect a site’s inclusion in Search. That a site which disallows Google-Extended can still appear in Google’s AI answers follows from those two documented statements — INFERRED, not quoted.

Separate controls do not always mean separate crawls. OpenAI: If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling. — DOCUMENTED · 2026-09-27.

CONTROLS THAT ARE NOT USER AGENTS

Page level: a meta value, a Search Console setting

Not every control is a line in robots.txt. These work at page level, without any user agent, and four providers document one. Microsoft’s two tags separate three states: in the AI answer with content (no tag), in the answer as URL, title and snippet only (NOCACHE), out of the answer entirely (NOARCHIVE) — and Microsoft states that both tags keep the page in ordinary search results.

The table scrolls sideways.

SIX MORE PROVIDERS

Mistral · Meta · Apple · Amazon · Common Crawl · ByteDance

Six providers that document their agents and are not in the tables above. Mistral documents a clean triple. Meta documents five agents — a search-class agent, a user fetcher, a multi-purpose crawler and two that do no AI-search job at all, for ads and for link previews — read from the rendered page, because the raw fetch is refused, and marked as such in every row. Amazon documents three agents with independent settings. Common Crawl is neither a search engine nor an assistant. ByteDance documents its crawler in Chinese; the quote stays in the provider’s language, our translation stands beside it.

Three agents here are marked multi-purpose: one agent, several documented uses, training among them — Applebot, Meta-ExternalAgent and Amazonbot. Only Apple gives the training use a control of its own, the token Applebot-Extended. The label is our framing, like the three jobs; each row’s note names the uses its provider documents.

The table scrolls sideways.

WHAT IS NOT DOCUMENTED

One name that circulates, and no provider source for it

One crawler name comes up constantly and appears in no document by its provider. It is UNVERIFIED here, and it stays that way until the provider documents it. Third-party directories were found and refused as sources — several exist, and none is a provider source.

Version 1.0 listed a second name here: ByteDance’s Bytespider. ByteDance does document it, in Chinese — it now has a row in the six-provider table, and the correction is recorded under re-verification.

User-Agent: *
Allow: /

Sitemap: https://www.deepseek.com/sitemap.xml

The table scrolls sideways.

Also not on this page

• A recommended robots.txt. What to allow is a business decision about your content, not a technical default. This page gives you the classes so you can make it.
• Any claim that allowing an agent produces a citation. Every provider that addresses the question declines to guarantee inclusion. Eligibility is not selection.
• Crawler lists compiled by third parties. Several exist; none is a provider source, and this page has one sourcing rule: a DOCUMENTED row comes from the provider.
• Server-log observations. We have not published crawler-hit logs. When we do, those rows will be OBSERVED and clearly separated from these.
• Google’s other crawlers on the same page — the image, video and news variants of Googlebot, Storebot-Google, Google-InspectionTool, GoogleOther and Google-CloudVertexBot. None is described there as feeding Google Search’s AI features; Google-CloudVertexBot crawls only when a site owner asks, for building Vertex AI Agents. The Google source link carries the full list.

LLMS.TXT

Google’s words, twice — and the file we maintain anyway

Google Search ignores it. In Google’s words:

You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them. Google · DOCUMENTED · 2026-09-27

It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them. Google · DOCUMENTED · 2026-09-27

That is the whole Google answer, and this page does not soften it. Which tools do read the file? Some do. This page does not list them, because we have not verified a single one from a provider source — any list here would be UNVERIFIED dressed as a reference. If we verify one, it gets a row, a link and a date like everything else.

We publish an llms.txt on this site and maintain it by hand, for the tools that read it. It is not part of any claim about Google.

Three neighbouring claims, same source, same date

The table scrolls sideways.

WHAT WE RUN ON OUR OWN SITE

Disclosure · OBSERVED · checkable against the live file

blackoarstudio.com/robots.txt · read 2026-09-27

User-agent: *
Allow: /

Sitemap: https://blackoarstudio.com/sitemap-index.xml

Every agent on this page is currently allowed on this site, including the training bots. That is a deliberate position for a studio with no licensed archive to protect, and it is not advice. A publisher with content worth licensing might reasonably decide the opposite — the training table above is the table that lets them do so without losing search visibility, where the provider documents the separation.

OBSERVED — a live capture of our own file, 2026-09-27. Not a provider statement.

The file is not the whole picture — see the CDN layer under verification. On the same date we read this site’s Cloudflare settings. No AI-crawler policy is set: Search, Agent and Training are left unblocked on every page, none of them is set to block only on pages with ads, the legacy “Block AI bots” rule is off, and so is Bot Fight Mode. Cloudflare does not manage the robots.txt above.

OBSERVED — read from our own account, 2026-09-27. Unlike the file, it cannot be checked from outside the account; it is recorded so that the robots.txt above is not taken for the whole configuration.

RE-VERIFICATION SCHEDULE

Next: 2026-12-05

This page describes other people’s systems and will go out of date without warning — in a way a broken-link check will not catch. Our own source list was compiled on 26 August 2026 and re-read on 2026-09-05 for version 1.0. What that re-read found — OBSERVED:

• Every URL on our own list of 26 August 2026 still resolved on 5 September 2026. A link check would have reported nothing.
• In the same ten days, two agents appeared in provider documentation that were on no list of ours: OpenAI’s OAI-AdsBot and Meta’s Meta-WebIndexer. Neither moved a URL.
• Google’s guide became explicit enough on llms.txt to quote verbatim. The URL did not change; the words did.
• Two documentation homes moved (OpenAI’s bots page and Google’s crawler overview), both with a redirect. Both are recorded under their current address here.

Version 1.1 — every source re-read on 2026-09-27

Ahead of schedule, for two reasons: Cloudflare announced new defaults for AI crawlers, and a review of version 1.0 against the providers’ documents found gaps. Every quotation was checked by script against the current document, every endpoint fetched, and all of it preserved with checksums. What had moved at the providers — OBSERVED:

• Every quoted passage still stands in its source. One stood under the wrong agent — described in the next line.
• The robots.txt quotation for Meta-ExternalFetcher in version 1.0, “might bypass robots.txt”, stands in Meta’s document only in the sentence about FacebookExternalHit. For Meta-ExternalFetcher the document says “may bypass robots.txt”. Meta dates its page 21 May 2026, so the likelier reading is a transcription error in version 1.0 rather than a change at Meta; the version-1.0 copy was a transcription, so it cannot be settled.
• Two sources moved, both still reached from the old address in a browser: Meta’s crawler page (from /docs/ to /documentation/) and Bing’s 2023 blog post (from /september-2023/ to /2023/9/). Both are listed under their current address, with the old one noted.
• All endpoints answered HTTP 200 with an address list in the body. The lists changed length, as lists do.
• Meta still refuses a raw fetch (HTTP 400), and Bing’s crawler table is still rendered by JavaScript. This time the rendered copies were preserved as well.
• The largest change in the period was not in a crawler document: Cloudflare scheduled new defaults for AI crawlers for 15 September 2026 (the CDN layer, under verification).

What we changed on this page — including what version 1.0 had simply missed:

• Corrected: ByteDance documents Bytespider — in Chinese, on the webmaster platform of its Toutiao search engine, with a reverse-DNS pattern and ten IP ranges. Version 1.0 listed it as undocumented. It now has a row, quoted in the provider’s language, and a verification entry.
• Added: CCBot to the verification table — Common Crawl’s reverse-DNS pattern and IP list. Both were in the document we read on 5 September; version 1.0 missed them.
• Added: which Amazon list belongs to which agent. The document said so on 5 September; version 1.0 listed the three pages without saying which is which, and their names mislead.
• Relabelled: Applebot, Meta-ExternalAgent and Amazonbot as multi-purpose — one agent, several documented uses, training among them. Version 1.0 filed Applebot under search and the other two under training. The Applebot and Amazonbot quotes are unchanged; Meta-ExternalAgent is now quoted to the end of its sentence, which names the second use.
• Added: Meta-ExternalAds and FacebookExternalHit, the two Meta agents that do no AI-search job, so that their names can be placed in a log.
• Added: where no verification endpoint is published — Meta for all five of its agents, Mistral for its training crawler.
• Added: the CDN layer — Cloudflare’s AI-crawler defaults, a note on the Googlebot and bingbot rows, and what this site runs at that layer.

Content drifts under stable URLs. That is why every row carries a date rather than just a link.

• Every row carries the date its source was last read. That date is the claim.
• Re-verified quarterly, and whenever a provider announces a change.
• When a source moves, changes its wording or disappears, the row is marked and re-dated. The old claim is never silently edited — the same rule we apply to our own measurements.
• Next scheduled re-verification: 2026-12-05.

SOURCES

Provider documents only · read 2026-09-27 · preserved with checksums

• OpenAI · crawler documentation developers.openai.com/api/docs/bots
• Google · common crawlers developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Google · user-triggered fetchers developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
• Google · AI features and your website developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Bing · which crawlers does Bing use www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 crawler table is JavaScript-rendered — read from the rendered page
• Bing Webmaster Blog · NOCACHE and NOARCHIVE blogs.bing.com/webmaster/2023/9/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat/ moved from blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat — noted 2026-09-27
• Perplexity · crawlers docs.perplexity.ai/docs/resources/perplexity-crawlers
• Anthropic · does Anthropic crawl data from the web support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Mistral · robots docs.mistral.ai/robots
• Meta · web crawlers developers.facebook.com/documentation/sharing/webmasters/web-crawlers moved from developers.facebook.com/docs/sharing/webmasters/web-crawlers/ — noted 2026-09-27 raw fetch refused (HTTP 400) — read from the rendered page
• Apple · about Applebot support.apple.com/en-us/119829
• Amazon · Amazonbot developer.amazon.com/amazonbot
• Common Crawl · CCBot commoncrawl.org/ccbot
• ByteDance · about Bytespider (Toutiao Search, in Chinese) zhanzhang.toutiao.com/docs/intro/26899
• Cloudflare · AI bot policies (block AI bots) developers.cloudflare.com/bots/additional-configurations/block-ai-bots/
• Cloudflare Blog · new AI traffic options blog.cloudflare.com/content-independence-day-ai-options/

Two documents could not be captured raw and are marked above and in their rows. Both were read from the rendered page; since version 1.1 the rendered copy is preserved as well. Recorded so a reader knows which rows rest on a preserved document and which on a transcription. The IP endpoints in the verification table were fetched on the same date and preserved alongside the documents. One document is in Chinese (ByteDance) and is quoted in Chinese.