Cloudflare's New Crawl Function: A Gift for Developers, a Nightmare fo

80% of Companies Can Use AI. Only 30% Do. Here's the 50-Poi… · Mar 14, 2026 Technology

Cloudflare sits between every website and the internet. Their new crawl function lets developers scrape virtually any site legally and without proxy headaches — bypassing even robots.txt restrictions. It's a massive unlock for AI builders. But for anyone who publishes content online and doesn't want it scraped, it's an unenforceable nightmare.