Cloudflare Just Started Blocking AI Crawlers by Default - What It Means for Your Site's AI Visibility

As of today, September 15, 2026, Cloudflare's default settings block "mixed-use" AI crawlers from crawling any page that carries ads - automatically, for new customers, new sites, and every existing free-tier site, unless the owner changes it (TechCrunch, July 1, 2026; Cloudflare developer docs, September 2026). It's the most consequential default change to how AI systems see the web since robots.txt started mattering for LLMs at all, and most site owners will not notice the moment it happens to them.

What actually counts as "mixed-use"

The labs already split this out on their own: OpenAI runs GPTBot for model training and a separate OAI-SearchBot for live ChatGPT search, while Anthropic runs ClaudeBot for training and Claude-SearchBot for its own search and answer features. A "mixed-use" crawler is one whose single user agent blends more than one of three distinct jobs into a bot Cloudflare can't cleanly categorize.

• Traditional search indexing - the job Googlebot has always done

• Agentic or answer fetching - an assistant reading a page live to answer a user's question

• Model training - bulk ingestion for a future model's dataset

A bot that declares one clean purpose is unaffected. A bot that blends two or three of those jobs into one identity is what gets blocked by default on ad-monetized pages starting today.

The business model shift behind the policy

Cloudflare is moving from its earlier "Pay Per Crawl" toll model to "Pay Per Use": publishers get paid when their content is actually used inside a generated answer, not merely fetched. Ceramic.ai and You.com are the first two partners live under this model (Cloudflare developer docs, September 2026).

For a smaller business, the tool that matters day to day is AI Crawl Control - free on every Cloudflare site, no configuration required - which shows every AI bot hitting the site and lets the owner allow, block, or charge each one individually, instead of inheriting one blanket default that may not fit the business at all.

Why this matters even if you don't sell content

This isn't only a publisher-economics story. If a prospective client asks an AI assistant "who does [X] well" or researches a vendor through ChatGPT, Perplexity, or Claude before ever visiting the site directly, being invisible to the search and agentic bots - not the training bots - costs qualified traffic that never shows up in an analytics dashboard, because the visitor arrived through an answer, not a click.

The common gut reaction to this story is to block all AI bots outright. That also blocks the search and answer crawlers a business actually wants - OAI-SearchBot, Claude-SearchBot, and equivalents from other assistants - not just the training-only ones. The useful response is a bot-by-bot allow list, not a blanket toggle.

What to actually do about it this week

1. Check Cloudflare AI Crawl Control (or your CDN's equivalent) today if the site runs behind Cloudflare - confirm which bots are currently allowed versus blocked, and don't assume the pre-September default still applies.

2. Explicitly allow the *-SearchBot and answer-agent crawlers even if you choose to block bulk training crawlers - they serve different purposes, and one blanket rule usually gets this wrong.

3. Publish or update an llms.txt file at the site root - a plain-language summary of what the business does and which pages matter most, which several assistants already use to ground answers faster than an unstructured crawl.

4. Re-check this quarterly - both crawler identities and Cloudflare's default policy are still moving targets in 2026.

We already ship an llms.txt alongside the sites we build, for exactly this reason - not because it guarantees a citation, but because the alternative, being unreadable to the systems more people now use to research a vendor before they ever click a link, is a worse and entirely avoidable default. Today's change is a good prompt to go check what your own site is actually letting through.