Recently, a dispute emerged between Cloudflare—a major internet infrastructure provider—and Perplexity, an AI-powered search and Q&A platform. At the center of the controversy is the question: What counts as a bot in the age of AI assistants? Here’s a breakdown of what Perplexity claims in response to Cloudflare’s accusations.
What Cloudflare Alleged
Cloudflare accused Perplexity of:
- Engaging in “stealth crawling” that bypassed robots.txt rules
- Using hidden bots and impersonation tactics to scrape websites
- Generating 20–25 million daily requests under suspicious behavior patterns
Cloudflare published a blog post outlining these concerns, including a technical diagram that supposedly explained how Perplexity’s system operated.
Perplexity’s Response, Summarized
In a detailed response, the Perplexity team offered a very different picture of how their system works.
1. User-driven Agents, Not Crawlers
Perplexity says it doesn’t use traditional web crawlers to index the internet. Instead, its system performs real-time content fetching only when a user asks a specific question. For example, when someone asks, “What’s the latest on that new phone release?”, Perplexity fetches relevant content in real time, summarizes it, and returns the result.
The company emphasizes that this process:
- Is initiated by real user queries
- Doesn’t store the fetched data long-term
- Isn’t used to train AI models
2. Not 25 Million Requests
Perplexity claims that the large volumes of web traffic Cloudflare observed were misattributed. According to them, the majority of the traffic—3–6 million daily requests—originates from BrowserBase, a third-party cloud browser service.
Perplexity says it uses BrowserBase only for specific, limited tasks, resulting in fewer than 45,000 daily requests. The company suggests that Cloudflare confused BrowserBase traffic (from many clients) with Perplexity’s own.
3. Diagram Called Inaccurate
Cloudflare’s blog included a diagram describing Perplexity’s “crawling workflow.” Perplexity responded by saying the diagram does not accurately represent how their systems function and bears no resemblance to their actual data flow or architecture.
4. Lack of Transparency from Cloudflare
Perplexity also stated that they had reached out to Cloudflare to understand the traffic analysis but didn’t receive answers. This, they say, left them with two possible explanations for the accusations:
- Cloudflare made a publicity-driven move and used Perplexity’s name for attention, or
- There was a technical failure in traffic attribution
Either way, Perplexity views the analysis as flawed and believes the claims were factually incorrect.
Why This Matters
The exchange raises broader questions about how infrastructure providers distinguish between:
- Traditional bots and scrapers
- Real-time, user-initiated agents
- AI assistants acting on behalf of individual users
Perplexity warns that mischaracterizing AI agents as bots could lead to overblocking and a “two-tiered internet,” where access to information depends more on the tool being used than the person seeking it.
They argue that if services like theirs are blocked, it could limit people’s ability to:
- Research personal or medical topics
- Compare product reviews
- Access timely news
Final Thought
Perplexity’s response presents an alternative perspective on what’s happening under the hood of modern AI platforms. Whether their explanation is accepted or not, the conversation highlights the need for clearer standards around web traffic, transparency in bot detection systems, and a deeper understanding of how AI tools interact with the open web.
Disclaimer: This article summarizes public statements made by the parties involved. AIholics does not take a position on the accuracy of either Cloudflare’s claims or Perplexity’s response.
