# Cloudflare Sets September 15 Deadline for AI Crawler Separation

> Cloudflare has set a September 15 deadline that could reshape how AI companies gather training data. From that date it will, by default, block crawlers that lump together model-training and search functions from reaching ad-hosting pages, applying the rule to new customers, existing sites and every free-tier account. The message to AI firms is blunt: separate your training bots from your search bots, or lose access to a large slice of the open web. CEO Matthew Prince framed the move around a milestone, noting that the majority of internet traffic is now non-human as bots have overtaken people online. Cloudflare is pairing the block with an evolution of its Pay Per Crawl idea into a Pay Per Use model, where publishers can charge when their content actually creates value, with Ceramic.ai and You.com among the first partners. For a web where publishers have watched AI models ingest their work for free, this is one of the more concrete attempts yet to put a tollbooth on the data pipeline, and its success or failure will shape whether scraping stays a free lunch.

_Section: [Daily AI Updates](https://www.wortins.com/daily-ai) · Source: TechCrunch · Published Thursday, July 9, 2026_

## Wortins' read

Cloudflare has set a September 15 deadline that could reshape how AI companies gather training data. From that date it will, by default, block crawlers that lump together model-training and search functions from reaching ad-hosting pages, applying the rule to new customers, existing sites and every free-tier account. The message to AI firms is blunt: separate your training bots from your search bots, or lose access to a large slice of the open web. CEO Matthew Prince framed the move around a milestone, noting that the majority of internet traffic is now non-human as bots have overtaken people online. Cloudflare is pairing the block with an evolution of its Pay Per Crawl idea into a Pay Per Use model, where publishers can charge when their content actually creates value, with Ceramic.ai and You.com among the first partners. For a web where publishers have watched AI models ingest their work for free, this is one of the more concrete attempts yet to put a tollbooth on the data pipeline, and its success or failure will shape whether scraping stays a free lunch.

## Source

[Read the full story at TechCrunch](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/)

## Related coverage

- [Gemini 3.5 Pro Delayed from June to July for Full Architectural Rebuild](https://www.wortins.com/story/gemini-3-5-pro-delayed-from-june-to-july-for-full-architectu-8ba8a785) — [Tech Insider](https://tech-insider.org/au/gemini-3-5-pro-delayed-july-2026/)
- [Why Infrastructure, Not Models, Determines AI Competitive Advantage](https://www.wortins.com/story/why-infrastructure-not-models-determines-ai-competitive-adva-edadc5e3) — [Apptad](https://apptad.com/insights/ai-data-what-actually-creates-competitive-advantage-in-2026/)
- [Perplexity Makes Its Comet AI Browser Free on Every Platform](https://www.wortins.com/story/perplexity-makes-its-comet-ai-browser-free-on-every-platform-3899f5d7) — [IBM Think](https://www.ibm.com/think/news/comet-perplexity-take-agentic-browser)
- [Taktile Raises $110M Series C to Automate Banking Decisions](https://www.wortins.com/story/taktile-raises-110m-series-c-to-automate-banking-decisions-b5bb1beb) — [Fortune](https://fortune.com/2026/06/24/exclusive-taktile-goldman-sachs-ai-bank-insurance-funding/)
- [Worldmodeldata Raises £7M Seed Round](https://www.wortins.com/story/worldmodeldata-raises-7m-seed-round-68be32c6) — [The Next Web](https://thenextweb.com/news/worldmodeldata-7m-seed-gaming-world-models)
- [China Invests $900M in AI Chip Champion Amid Competition](https://www.wortins.com/story/china-invests-900m-in-ai-chip-champion-amid-competition-fc01a53d) — [BigGo Finance](https://finance.biggo.com/news/6f0c6bb2-795f-4c57-9d09-6db691d7638a)

---

_Curated and written by [Wortins](https://www.wortins.com) — The daily AI briefing. Every story links to its original source; the "Wortins read" on each is our own original analysis. [About Wortins & our editorial approach](https://www.wortins.com/about)._