Cloudflare flipped a switch on September 15, 2026, and the default behavior of the internet’s AI bots changed overnight. Every new domain that onboards to the network now sorts incoming crawlers into three buckets, Search, AI Training, and AI Agent, and two of those three get blocked by default the moment a page carries ads. Cloudflare re-evaluated every page under the new rules the very next day, September 16. The move caps a run of AI-crawler policy changes that started with a blog post in July and landed, in full, in the middle of September.
The timing matters. Cloudflare sits in front of a large share of the web’s traffic, so a default-on-block change at the edge affects far more sites than any single publisher’s robots.txt file ever could. It also arrives alongside a parallel push: Cloudflare wants to let publishers get paid when a bot does cross the line, not just blocked when it doesn’t. Pay Per Crawl, Monetization Gateways, and a new “Cloudflare Wallets” product turn the crawler fight from a pure access-control problem into a billing problem. This piece breaks down what changed, why Cloudflare picked this particular week to ship it, how it stacks up against what Google, Reddit, and TollBit are doing, and what site owners and developers should check before the next default shift lands.
What Changed On September 15, 2026
Cloudflare had signaled the shape of this change back on July 1, 2026, in a changelog post describing new AI traffic controls. The actual switch flipped on September 15. According to Cloudflare’s own developer documentation, “On September 15, 2026, we’ll be setting new defaults for each of these three classifications,” referring to how the network buckets automated traffic (Cloudflare). New domains that connect to Cloudflare after that date inherit updated defaults out of the box, without an administrator having to flip anything manually.
The practical effect: bots that Cloudflare classifies as Training or Agent are now blocked by default on pages that display ads, while bots classified as Search stay allowed. That single distinction, ad-supported page versus non-ad page, determines whether a crawler gets through on day one. Cloudflare’s changelog spells it out directly: “Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed” (Cloudflare). Independent analysis of the rollout confirmed the date lines up with what Cloudflare announced two and a half months earlier: “Cloudflare announced the shape of this change on July 1 and switched it on September 15,” and by the following day “every page was re-evaluated” under the new ruleset (DigitalApplied).
Existing domains that were already on Cloudflare before September 15 were not forcibly reset to the new defaults, based on Cloudflare’s own framing of the change as applying to “new domains onboarding.” That distinction matters for any site that configured its bot rules manually before the update landed, since those custom settings remain untouched unless the operator changes them.
Search, Training, And Agent: The Three New Categories
Cloudflare’s system does not treat “AI bot” as one label. It splits automated traffic into three behaviors that site owners can now control independently. Per Cloudflare’s own documentation, “Cloudflare groups AI traffic into three behaviors you can control independently: Search, Agent, and Training” (Cloudflare).
Search covers crawlers that index content for retrieval or citation in AI-generated answers, the category Cloudflare treats most like a traditional search engine spider. Training covers bots whose purpose is pulling content into a model’s training data, the category most publishers actually want blocked. Agent covers bots acting on behalf of a live user session, fetching a page because a person asked an AI assistant to check something in real time. That third bucket is the newest and the trickiest, because an agent request looks a lot like a human visit but originates from a data center rather than a browser.
Splitting the traffic this way lets a site block Training while still allowing Search, a combination that was previously an all-or-nothing choice on most networks. Cloudflare’s own September 15 press messaging framed the entire release around removing that binary choice, positioning the update as letting publishers end the search-or-AI-training tradeoff rather than forcing them to pick one.
Inside The 20-Bot Census: Who Gets Blocked
An independent audit published by DigitalApplied tracked 20 named AI bots against Cloudflare’s new classification system to see how each one landed after the September 15 switch. The analysis found that the census covers the full range of automated traffic types Cloudflare now recognizes, from well-known model-training crawlers to newer agent-style bots that fetch pages on a user’s behalf (DigitalApplied). The exact list of 20 bots and their individual classifications gives site operators a reference point for checking whether a specific crawler they care about, whether it is helping their SEO or hurting their content licensing leverage, falls on the allowed or blocked side of the new default.
The census approach matters because Cloudflare’s classification scheme is not static. New bots enter the ecosystem regularly as AI labs ship new agents and crawlers, and Cloudflare has to decide where each one lands: Search, Training, Agent, or some mixed-use category that does not fit cleanly into any single bucket. That mixed-use problem is exactly what Cloudflare’s Accountable crawler standard, covered below, was built to address.
Disallow AI Training And Bot Preference Sync
Alongside the new defaults, Cloudflare shipped a feature called Bot Preference Sync, which automatically publishes a site owner’s crawling preferences into the standard robots.txt file so that compliant bots outside Cloudflare’s own network still see the same instructions. Per Cloudflare’s own description, the second option in its new AI controls reads: “2. Disallow AI Training: Bot Preference Sync publishes the applicable no-training preference in robots.txt” (Cloudflare).
That sync matters because network-level blocking at Cloudflare’s edge only stops crawlers that respect Cloudflare’s fingerprinting and rate limits. A bot running from a residential proxy or a rotating IP pool can sometimes slip past edge-level detection. Publishing the same preference into robots.txt gives a second, machine-readable signal that any crawler operator claiming to be compliant has to check, and it creates a paper trail if a company’s bot ignores a documented no-training preference.
Below is a simplified illustration of how a no-training preference typically appears once synced to robots.txt. Exact syntax varies by crawler and by how a given site has configured its Cloudflare dashboard.
# Illustrative robots.txt entry after Bot Preference Sync
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Googlebot
Allow: /The Accountable Crawler Standard
Not every AI-related bot fits neatly into Search, Training, or Agent. Some crawlers do double duty, indexing content for a chatbot’s live citations one moment and feeding a training pipeline the next. Cloudflare’s response was to define a category of Accountable crawlers, bots that disclose their purpose clearly enough that a site owner can apply the right rule to the right traffic instead of guessing. Cloudflare has also said it is participating in broader industry efforts to standardize how AI systems declare their access to web content, an effort aimed at making the current patchwork of company-specific bot names and behaviors easier to police at scale.
The practical upside for site operators is fewer false positives. A crawler that honestly labels itself as mixed-use, part search index, part training pull, can be handled with a rule that matches its actual behavior rather than an operator having to choose between blocking it entirely or letting it through unchecked. Cloudflare’s own criteria for what counts as accountable were tied to giving site owners, in the company’s words, “a clear way to opt out of AI training through robots.txt or a comparable standard” as part of its September 15 rollout materials.
Pay Per Crawl, Monetization Gateways, And Wallets
Blocking is only half of Cloudflare’s September strategy. The other half is getting publishers paid when they choose to let a crawler through. Cloudflare has been building out a monetization stack alongside the access controls: Monetization Gateways to help content providers sell access to their material, Cloudflare Wallets to support machine-to-machine payments for autonomous agents, and Pay Per Crawl, which lets a publisher set a price for crawler access rather than a flat allow-or-deny rule.
The logic connects directly to the new default-block behavior. Once Training and Agent bots are blocked by default on ad-supported pages, a publisher has two real options beyond leaving the block in place: negotiate a direct licensing deal with the AI company running the bot, or turn on Pay Per Crawl and let the market set a price automatically. Cloudflare Wallets exist to make that second option work at machine speed, since a human sitting in the loop to approve every crawl request would defeat the purpose of an automated pricing system. DataForSEO keyword research shows meaningful and rising search interest in “cloudflare pay per crawl” among developers and site operators trying to understand how to configure it, a signal that adoption questions are already surfacing beyond Cloudflare’s own customer base.
Cloudflare’s Three AI Traffic Classifications
The table below summarizes how each classification behaves under the September 15, 2026 defaults, based on Cloudflare’s own changelog and blog documentation.
| Classification | Default On Ad-Supported Pages (Sept 15+) | What It Covers | Owner Controls |
|---|---|---|---|
| Search | Allowed | Indexing for retrieval and citation, closest to traditional search spiders | Can still be blocked manually per bot |
| AI Training | Blocked | Bots pulling content into model training pipelines | Disallow AI Training option, synced to robots.txt |
| AI Agent | Blocked | Bots acting on behalf of a live user session or assistant query | Can be allowed selectively, e.g. via Pay Per Crawl |
| Mixed-use / Accountable | Depends on disclosed behavior | Crawlers that disclose multiple purposes (search + training) | Rule applied per disclosed behavior, not a blanket setting |
A Crowded September: Cloudflare’s 2026 Rollout Timeline
The AI crawler default change did not happen in isolation. A tracker that logs Cloudflare’s product changelog counted 80 separate product updates across September 2026 alone, spanning AI controls, web application firewall rules, Workers limits, and client deprecations (SpyingBee). The table below lines up the changes most relevant to developers and site operators watching this specific story.
| Date (2026) | Change | Detail |
|---|---|---|
| Sept 1 | D1 free-tier row limits enforced | Daily read/write caps now apply to Workers Free plan D1 databases; queries fail for the rest of the day once exceeded |
| Sept 8 | Post-quantum TLS origin handshakes | Automatic Key Exchange rolled out across roughly 45 billion daily connections |
| Sept 10 | Post-quantum DNSSEC validation | 1.1.1.1 resolver begins validating ML-DSA-44 signatures, 2,420 bytes each |
| Sept 10 | Emergency WAF rule | Blocks an actively exploited Adobe Commerce/Magento RCE zero-day, CVE-2026-75650 |
| Sept 10 | Workflows retention default cut | Workers Paid plan default retention drops from 30 days to 7 days |
| Sept 15 | AI crawler defaults switch on | Search, Training, Agent classifications apply to new domains; ad pages block Training and Agent by default |
| Sept 15 | Fine-grained Workers access scoping | Teammates, CI tokens, and agents can be scoped to individual Workers instead of account-wide access |
| Sept 16 | Full re-evaluation | Every page on affected domains re-checked against the new AI traffic defaults |
| Sept 18 | cloudflared deprecation notice | 32-bit Windows and Intel-based macOS builds to stop receiving new releases starting 2027 |
Seen together, the pattern is a network hardening its edge on two fronts at once: cryptography moving to post-quantum defaults, and bot traffic moving to a pay-or-block-by-default model. Both shifts share a theme. Cloudflare is trying to make the secure or restrictive option the one that requires no configuration, and the permissive or legacy option the one an administrator has to opt into deliberately.
From Robots.txt To Paid Crawling: Historical Context
Robots.txt has governed crawler behavior since the mid-1990s, and it has always relied on voluntary compliance rather than enforcement. A crawler that ignores the file simply ignores it, and for most of the web’s history that was a minor problem because search engines had every incentive to follow the rules in exchange for being indexed. Generative AI broke that incentive structure. A model-training crawler gets no direct benefit from respecting a publisher’s wishes, so plenty of AI companies’ bots have been reported ignoring or ambiguously interpreting robots.txt directives, especially in the early wave of AI crawling activity in 2023 and 2024.
Cloudflare’s answer over the past two years has been to move enforcement from the honor-system file down to the network layer it already controls. Instead of trusting a bot to read and respect a text file, Cloudflare fingerprints traffic patterns, verifies bot identity claims, and applies rules at the edge before a request ever reaches origin infrastructure. The September 15 default change is the furthest extension of that strategy yet. It does not just give publishers a tool to block AI training, it flips the tool on by default for anyone who has not explicitly turned it off. That is a meaningful departure from the historical default of allow-unless-blocked, and it puts Cloudflare in the position of setting norms for a large share of the web rather than just enforcing whatever norm each individual site owner picks.
Competitive Landscape: Google, Reddit, TollBit, And Rivals
Cloudflare is not the only company trying to solve the AI-crawler problem, and its network-level, default-on approach is a genuinely different bet than what competitors have tried. Google has leaned on its own Google-Extended token, a robots.txt directive that lets a site block Google’s AI-training use of its content while still allowing normal Search indexing, effectively the same Search-versus-Training split Cloudflare now enforces, but implemented as an opt-in file edit rather than a network-level default.
Reddit has taken the licensing-deal route instead of the technical-control route, signing direct data-access agreements with AI companies rather than relying on crawler-level blocking to force a negotiation. TollBit occupies a similar space to Cloudflare’s Pay Per Crawl, offering a metered-access layer that publishers can drop in front of their content to charge AI companies per request, but TollBit operates as a specialized vendor rather than as the CDN and edge network already sitting in front of a large share of global web traffic. That distribution advantage is Cloudflare’s real edge in this fight: it does not need publishers to install anything new, because the enforcement point is infrastructure most large sites already run through.
The tradeoff is that Cloudflare’s approach only protects sites that use Cloudflare. A publisher on a different CDN, or with no CDN at all, still has to rely on the older, weaker robots.txt-only model unless that publisher also adopts a comparable network-level tool from another vendor.
What Site Owners And Developers Should Do Now
Existing Cloudflare customers should not assume the new defaults silently changed their configuration, since Cloudflare applied the update to newly onboarding domains rather than resetting every existing account. Anyone running a site that predates September 15 should check the AI Audit or bot management section of the Cloudflare dashboard directly to see which classification rules are currently active, rather than assume the new block-by-default behavior is already in place.
Developers building anything that fetches web content programmatically, including AI agents, research tools, or monitoring scripts, should check whether their traffic gets classified as Agent under Cloudflare’s new scheme, since that classification is blocked by default on ad-supported pages even when the underlying intent has nothing to do with model training. A legitimate uptime monitor or price-comparison tool built on an LLM-powered agent framework could get swept into the same default-block bucket as a training crawler if it is not properly identified. Registering the bot’s identity clearly, and where relevant applying for Accountable crawler status, is the practical fix.
Publishers weighing whether to turn on Pay Per Crawl should treat it as a revenue experiment rather than a guaranteed income stream. The tooling is new, adoption among AI companies willing to pay per request is still forming, and the near-term value for most sites will likely come more from the leverage of being able to say no by default than from actual crawl-fee revenue.
Market And Investor Reaction
Financial media picked up the story quickly. Coverage from September 15 through September 16 framed Cloudflare’s AI content controls as giving website owners real leverage over how their material gets used in AI training, tying the announcement directly to the company’s broader AI strategy narrative for investors (Simply Wall St). The framing matters for how the market reads Cloudflare’s positioning. Rather than being caught reacting to AI disruption of its core CDN and security business, Cloudflare is presenting itself as the infrastructure layer that arbitrates the relationship between publishers and AI companies, a position with recurring revenue potential if Pay Per Crawl and Monetization Gateways gain real adoption.
That narrative sits alongside Cloudflare’s separate push into post-quantum cryptography defaults earlier the same month, reinforcing an overall story of a company trying to own multiple layers of default security and default control at the edge, not just one. Investors and analysts tracking Cloudflare’s stock, ticker NET, have increasingly treated these edge-layer product announcements as leading indicators of the company’s ability to keep expanding average revenue per customer beyond its original CDN and DDoS-protection business.
What Cloudflare’s Own Documentation Says
Cloudflare has been unusually direct about both the mechanics and the intent behind this rollout, publishing the technical detail in its developer changelog rather than keeping it confined to marketing copy. On the underlying philosophy, Cloudflare’s changelog is explicit that the three-category system exists so that “Cloudflare groups AI traffic into three behaviors you can control independently: Search, Agent, and Training” (Cloudflare), a framing that treats crawler policy as a set of independent dials rather than a single on/off switch.
On the mechanics of how a no-training preference actually propagates beyond Cloudflare’s own edge, the company describes Bot Preference Sync as the bridge: “Disallow AI Training: Bot Preference Sync publishes the applicable no-training preference in robots.txt” (Cloudflare). And on what changed operationally for new customers on day one, the changelog states plainly that “Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed” (Cloudflare). Taken together, the three statements describe a company that built the classification system first, then flipped the default months later once the underlying mechanics had time to stabilize.
Five Predictions For AI Crawler Policy
- Other CDNs follow with their own default-block posture. Cloudflare’s scale gives it first-mover advantage in setting a new norm; competitors in the CDN and edge-security space will face pressure to match a default-block stance rather than ship it as an opt-in feature.
- Pay Per Crawl adoption grows unevenly. Large publishers with strong content leverage will negotiate direct deals or turn on metered access quickly, while smaller sites will mostly use the free block-by-default protection without engaging the monetization layer for months.
- The Agent category becomes the next flashpoint. As more consumer AI assistants fetch live pages on a user’s behalf, expect disputes over whether Agent traffic should really face the same default block as Training traffic, since it serves a live human request rather than a training pipeline.
- Accountable-crawler disclosure becomes a minimum bar for legitimate AI companies. Bots that refuse to disclose their classification clearly will find themselves increasingly blocked by default across more of the web, not just on Cloudflare.
- Regulatory scrutiny of AI training data sourcing increases. A network as large as Cloudflare setting default rules for AI data access will draw attention from policymakers already examining AI training data practices, particularly in the EU and among US state-level privacy regulators.
Frequently Asked Questions
What exactly changed on Cloudflare on September 15, 2026?
New domains onboarding to Cloudflare after that date received updated default rules for AI bots. Bots classified as AI Training or AI Agent are now blocked by default on any page that displays ads, while bots classified as Search remain allowed by default.
Does this affect websites that were already on Cloudflare before September 15?
Cloudflare framed the change as applying to new domains onboarding to the network. Existing sites should check their dashboard’s AI Audit or bot management settings directly rather than assume the new defaults were applied automatically to an established account.
What is Bot Preference Sync?
It is the Cloudflare feature that automatically publishes a site’s AI-training preference into the standard robots.txt file, so that crawlers outside Cloudflare’s own detection network still receive a machine-readable no-training signal.
What is Pay Per Crawl?
Pay Per Crawl is a Cloudflare monetization tool that lets a publisher set a price for AI crawler access instead of a simple allow-or-block rule, turning a blocked bot into a billable one if the publisher chooses to open access at a price.
What are Cloudflare Wallets?
Cloudflare Wallets are infrastructure for machine-to-machine payments, built to support scenarios like autonomous AI agents paying for crawl access or other automated services without a human approving each individual transaction.
What counts as an Accountable crawler?
Cloudflare defines Accountable crawlers as bots that clearly disclose their purpose, including mixed-use bots that both index for search and pull content for training, so that site owners can apply the correct rule instead of guessing at a bot’s intent.
How is the Agent category different from Training?
Training bots pull content into a model’s training pipeline. Agent bots fetch a page on behalf of a live user session, such as an AI assistant checking a webpage in response to a real-time question. Both are blocked by default on ad-supported pages, but they serve very different purposes.
How does Cloudflare’s approach compare to Google’s Google-Extended directive?
Google-Extended is an opt-in robots.txt directive a site owner has to add manually to block AI training while keeping Search indexing intact. Cloudflare’s system applies a similar Search-versus-Training split, but does it as a network-level default that new domains receive automatically rather than a file edit each site owner has to make themselves.
