Cloudflare AI Crawler Rules: Avoid Blocking Google in 2026

Why This Matters Right Now

On July 1, 2026, Cloudflare announced a significant overhaul of how they handle AI bot traffic — and set a deadline that every site owner on their network needs to know about.

September 15, 2026. That's when Cloudflare's new default settings kick in automatically for new domains and existing free-tier customers who haven't reviewed their settings. If you run a site on Cloudflare and do nothing before that date, your AI crawler policy will change whether you intended it to or not.

Here's the context that makes this urgent: Cloudflare CEO Matthew Prince recently confirmed that bot traffic has surpassed human traffic on the internet for the first time — a milestone he didn't expect until late 2027. AI crawlers now account for the majority of crawler requests on Cloudflare's network, up from roughly 20% in spring 2025. Daily AI agent requests alone increased by more than 1,700% over the past year.

The old system — a single "Block AI Bots" toggle — was built for a simpler world. That world no longer exists. Cloudflare's new system gives you three separate controls for three fundamentally different types of bots. Understanding the difference is what this guide is about.

This isn't just a settings walkthrough. By the end you'll know exactly what each bot type means for your site, what you should do based on your site type, and how to configure everything in your Cloudflare dashboard before September 15.

The 3 Types of AI Bots Cloudflare Now Separates

Cloudflare has replaced the single "Block AI Bots" toggle with three independent categories. Each one describes what a bot actually does on your site — not just whether it calls itself an AI.

1)Search Bots

What they do: Crawl and index your content so AI-powered search experiences can answer questions about it later. When someone searches on Perplexity, ChatGPT search, or even Google's AI Overviews, these bots are what made your content findable.

Value they return: Referral traffic, citations, and visibility in AI search results. This is the category that sends people back to your site.

Real examples: OAI-SearchBot (OpenAI's search crawler), PerplexityBot, ClaudeBot (Anthropic), Bingbot (for search indexing)

Cloudflare's default: Allowed by default — and this won't change on September 15.

2)Agent Bots

What they do: Act in real time on behalf of a specific user. When someone asks ChatGPT to "check this website for me" or uses a browser-use agent, that's an Agent bot visiting your site. It's not building an index — it's fetching content for one person, right now.

Value they return: Almost none directly. The person using the agent gets the content without visiting your site. No ad impression, no referral, no revenue.

Real examples: ChatGPT-User (OpenAI's browsing agent), browser automation agents powered by Gemini or Claude, autonomous research agents

Cloudflare's new default from September 15: Blocked on pages that display ads.

3)Training Bots

What they do: Scrape your content to train or fine-tune AI models. Your writing, your data, your original work — absorbed permanently into a model's weights. No referral traffic comes back. No compensation unless you set it up.

Value they return: Zero, unless you're in a paid licensing arrangement.

Real examples: GPTBot (OpenAI's training crawler), CCBot (Common Crawl, used by many model trainers), Google-Extended (Google's AI training crawler, separate from Googlebot), Meta-ExternalAgent

Cloudflare's new default from September 15: Blocked on pages that display ads.

The Key Insight

A single company can operate multiple bots across different categories. Google is the clearest example — and it's where most site owners will have their biggest confusion. More on that next.

The Googlebot Problem Nobody Is Explaining Clearly

This is the part of Cloudflare's update that has real consequences for your SEO, and most coverage has buried it in technical language. Let's be direct.

Googlebot is a multi-purpose crawler. It indexes your content for Google Search (that's a Search behavior) AND collects content for AI training — powering Google Gemini, AI Overviews, and AI Mode (that's a Training behavior). It does both in a single crawler under the same user agent.

Here's the problem: Cloudflare will now apply the most restrictive rule to any multi-purpose crawler.

That means:

• If you block Training bots → Googlebot gets blocked on your ad pages

• If you block Agent bots → Googlebot is fine (Googlebot isn't an agent)

• If you only block Training bots on ad pages → Googlebot is blocked on those specific pages

This takes effect September 15, 2026 for new domains and existing free customers who haven't changed their settings.

If you run an ad-supported site and you currently have the legacy "Block AI Bots" toggle enabled, Googlebot may already be configured to be blocked on your ad pages after September 15 — unless you opt out.

How to avoid this:

Google provides a separate crawler called Google-Extended specifically for AI training opt-outs. You can disallow Google-Extended in your robots.txt file, which tells Google not to use your content for AI training — without affecting Googlebot's ability to index you for search.

But there's a gap worth knowing about. Google-Extended stops your content from training Google's models, but it doesn't remove you from AI Overviews — those pull directly from Google's regular Search index, not a separate crawl. Right now there's no way to appear in AI Overviews while fully opting out of AI training use of that same content. Google hasn't solved this gap yet. So think of Google-Extended as a partial fix, not a complete one.

Your options in simple words are given below:

• Option A: Block Training bots in Cloudflare dashboard + add Disallow: / for Google-Extended in robots.txt. This blocks all training bots except Googlebot stays for search, and you've told Google separately not to use your content for AI training — though your content may still surface in AI Overviews through Search itself.

• Option B: Block Training bots on all pages except ad pages. Googlebot continues normally everywhere.

• Option C: Don't block Training bots at all, and rely on robots.txt + Google-Extended disallow for Google specifically.

The right option depends on your site type — see the decision table below.

Step-by-Step: How to Configure Cloudflare AI Crawl Controls

These controls are available on all Cloudflare plans including free. Here's exactly where to find them and what to do.

Step 1 — Navigate to AI Crawl Control

Log into your Cloudflare dashboard. Select your domain. Go to:

Security → Bots → AI Crawl Control

You'll land on a page with two tabs: Robots.txt and Crawlers.

Step 2 — Review the Robots.txt Tab

This shows you the robots.txt Cloudflare is serving on your behalf, including any Content Signals or block directives that have already been injected. Check:

• Is Cloudflare managing your robots.txt? If yes, it will show the Content Signals format with directives at the top.

• Are any bots already disallowed that you didn't intend?

• Does Google-Extended appear with a Disallow directive? If not and you want to block Google AI training, you'll need to add it.

Step 3 — Review the Crawlers Tab

This shows you every known AI crawler that has visited your site, with:

• Request counts per crawler

• Current allow/block status

• robots.txt compliance logs (whether bots are respecting your directives)

Scan this list. If you see OAI-SearchBot or PerplexityBot showing as blocked, that's a problem — those are search bots that send referral traffic, and blocking them hurts your AI search visibility.

Step 4 — Set Your Three Category Controls

Back in the main AI Crawl Control panel, you'll see three managed presets: Search, Agent, and Training.

For each, you have three options:

• Block on all pages — stops this bot type everywhere

• Block only on pages that display ads — stops them where your revenue is, lets them through elsewhere

• Do not block — allows all bots of this type

For most site owners the recommended starting configuration is:

Then add Google-Extended as a Disallow in your robots.txt separately to handle Google's AI training crawler without affecting Googlebot.

Step 5 — Check for WAF Rule Conflicts

Cloudflare's WAF (Web Application Firewall) rules are enforced before robots.txt is even read. If you have a WAF rule that challenges or blocks certain user agents, it can override your AI Crawl Control settings.

Navigate to Security → WAF → Custom Rules and check for any rules affecting AI crawler user agents. If Perplexity or OAI-SearchBot are being hit with JS challenges, create explicit "skip" rules for their verified user agents.

Step 6 — For Each Specific Crawler

In the Crawlers tab, you can set individual bot-level rules on top of the category presets. For each crawler, your options are:

• Allow — crawler can access your content

• Block — crawler is fully denied

• Charge — triggers Cloudflare's Pay Per Crawl/Use system (more on this below)

• Enforce robots.txt — respects your robots.txt directives for this specific bot

Use individual rules when you want more granularity than the three category presets allow. For example: allow OAI-SearchBot (OpenAI's search crawler) but block GPTBot (OpenAI's training crawler) — these are separate bots, so you can configure them independently.

What Should YOUR Site Do? — Decision Table by Site Type

There's no single right answer. Here's a practical framework based on what your site is and what you're trying to protect.

Key principle: Search bots almost always stay allowed — they're what keeps you visible in AI search and traditional search. The real decision is about Agent and Training bots, and it comes down to whether you want human eyeballs on your ad pages or not.

The Pay Per Use Angle — Can You Actually Earn From AI Crawlers?

Yes — and this is the part most how-to guides aren't covering.

Cloudflare launched what they're calling the Monetization Gateway, building on their earlier Pay Per Crawl experiment. The new model is called Pay Per Use — and it's a meaningful step forward.

The difference: Pay Per Crawl charged AI companies based on whether they visited your page. Pay Per Use charges based on how they actually use your content — specifically, when it appears in an AI chatbot's answer.

In practical terms: if GPT-4o uses your article to answer someone's question, you get paid for that usage — not just for the crawl that collected it.

Current state as of July 2026:

• Cloudflare has opened a waitlist for the Monetization Gateway

• Early partnerships include Ceramic.AI and You.com

• Payments settle in stablecoins via the x402 open protocol — no payments infrastructure needed on your end

• Enterprise customers also get Attribution Business Insights, a dashboard showing which crawlers actually refer traffic back vs. just extract

What this means for you practically right now:

The Monetization Gateway is in early access — it's not yet a meaningful revenue stream for most sites. But the infrastructure is being built, and joining the waitlist positions you to benefit when larger AI platforms adopt it.

For now, the bigger opportunity is using the controls to protect your ad revenue first, and treating the monetization gateway as a future upside.

To join the waitlist: log into your Cloudflare dashboard → look for the Monetization Gateway option under Security → Bots, or visit the Cloudflare blog directly to find the waitlist link.

What to Do Before September 15 — Your Checklist

You have until September 15, 2026. Here's the exact sequence to work through:

✅ Action 1 — Audit your current settings Go to Security → Bots → AI Crawl Control. Check whether you have the legacy "Block AI Bots" toggle enabled. If yes, your site is already affected by the September 15 multi-purpose crawler change — Googlebot will be blocked on your ad pages unless you act.

✅ Action 2 — Check the Crawlers tab for blocked search bots Look for OAI-SearchBot, PerplexityBot, and ClaudeBot (Anthropic's search crawler). If any are showing as blocked, change them to Allow — these send referral traffic and blocking them hurts your AI search visibility.

✅ Action 3 — Set your three category controls Use the decision table above to set Search, Agent, and Training presets appropriate for your site type. Most sites: Search = Allow, Agent = Block on ad pages, Training = Block on ad pages.

✅ Action 4 — Handle Google separately via robots.txt Add a Disallow: / directive for Google-Extended in your robots.txt. This tells Google not to use your content for AI training without affecting Googlebot for search indexing. This is the clean way to handle the Googlebot multi-purpose problem.

✅ Action 5 — Check for WAF conflicts Review Security → WAF → Custom Rules for any rules that might challenge legitimate search crawlers. Create skip rules for verified AI search bot user agents if needed.

Opt-out option: If you want to keep your current behavior exactly as it is — no changes on September 15 — go to your Security settings and mark the opt-out before the deadline. Cloudflare will also notify customers as September 15 approaches.

AIWerse Verdict

Cloudflare has built the most practical set of AI crawler controls available to site owners — and made them free. The September 15 deadline makes this urgent, but the controls themselves are genuinely useful regardless of the deadline.

The move to take right now: spend 15 minutes in your Cloudflare dashboard going through the five-step checklist above. Most site owners should end up with Search bots allowed, Agent and Training bots blocked on ad pages, and Google-Extended disallowed in robots.txt. That's a configuration that keeps you visible in AI search while protecting the revenue that funds your content.

The bigger picture: Cloudflare is positioning itself as the policy and payment layer between content owners and AI companies. That's a significant shift from just being infrastructure. Whether the monetization side delivers for publishers depends on AI platform adoption — but the control side is real and available today.

Your content has value. Now you have the tools to decide who gets it and on what terms.

FAQs

Will Cloudflare block Googlebot if I block training bots?

Yes — potentially on your ad-monetized pages, starting September 15. Googlebot crawls for both search indexing and AI training, so Cloudflare applies its Training block rule to it. To avoid this, either opt out of the September 15 defaults, or use Google-Extended in your robots.txt to signal your AI training preferences to Google separately while keeping Googlebot allowed in Cloudflare.

Is Cloudflare's AI Crawl Control available on the free plan?

Yes. All three category controls — Search, Agent, and Training — are available to all Cloudflare customers including free tier. BotBase (the detailed bot directory) and Attribution Business Insights are Enterprise Bot Management features, but the core controls you need to set your policy are free.

What is Cloudflare BotBase?

BotBase is Cloudflare's searchable database of all known bots and agents. It shows how each bot is classified by behavior (Search, Agent, Training, and others), whether it's Verified, and its crawl-to-referral ratio. It's available to Enterprise Bot Management customers and is useful for understanding exactly which bots are visiting your site and why.

What's the difference between robots.txt and Cloudflare AI Crawl Control?

robots.txt is a voluntary signal — bots are supposed to respect it, but there's no enforcement mechanism. Cloudflare AI Crawl Control enforces rules at the network level, before a bot can even read your robots.txt. WAF rules take priority over both. The recommended approach is to use both: robots.txt for signaling (especially for Google-Extended), and Cloudflare controls for actual enforcement.

Should I block all AI bots to protect my content?

Not if you want to stay visible in AI search. Blocking all AI bots — including search crawlers — removes your content from Perplexity, ChatGPT search, and AI-enhanced results. The smarter move is to block Training and Agent bots (which don't return traffic) while keeping Search bots allowed (which do).

What is Cloudflare's Pay Per Use and should I sign up?

Pay Per Use is Cloudflare's framework for compensating content owners when their work appears in AI chatbot answers — not just when crawled. It's currently in early access with limited AI platform partnerships. Worth joining the waitlist, but don't count on meaningful revenue from it in 2026. Protect your ad revenue first via the bot controls; treat Pay Per Use as a future upside.

Does this affect my existing Cloudflare setup if I'm not on a free plan?

Yes. The September 15 changes apply to new domains across all plans and to existing free customers who haven't changed their settings. Paid plan customers with existing custom configurations are less likely to be auto-changed, but you should still audit your settings and confirm your current bot policy is intentional.

Related News & Updates

• Claude Opus 4.8 is launched — What's Actually New?

• Cursor Just Got Smarter — What's New in Cursor 3.7 update?

• AI Comparisons

Junaid Nawaz is the founder of AIwerse and a developer focused on AI tools, agentic workflows, and builder-focused tech. He covers AI model releases, coding tools, and platform updates for developers and teams building with AI. You can follow AIwerse on X (@AIwerse).