Why AI Agents Get Blocked on the Web (and How to Fix It)

Your AI agent fails to read a page not because it's lazy, but because a growing share of the web is designed to turn it away. Cloudflare's Radar data crossed a milestone in June 2026: bots now generate roughly 57% of web requests to HTML content, versus 42% from humans — and sites have responded by blocking automated traffic harder than ever. If your agent keeps hitting 403s, captchas, or paywalls, the problem is usually bot detection, not a bad prompt.

First, the number everyone misquotes

The crossover is real but narrower than the headlines suggest. Cloudflare measured that about 57.4% of requests to HTML content on sites it hosts come from bots, with humans at 42.6%. That's automated requests to a sample of websites it protects — not "most internet traffic is AI" across the whole web, and not "humans are gone." Tools like Imperva and HUMAN report a lower bot share when measuring differently. The takeaway: bots are a large and growing chunk of web traffic, and sites are responding by locking down.

That matters to you because it created a genuine arms race. As agents browse more, sites deploy thicker bot detection, and owners of agents need a way to work around it.

Bots vs humans on the open web: Cloudflare Radar, June 2026

Why sites block agents (it's not all paranoia)

There are four real reasons, and understanding them tells you which fix applies to your problem:

1. Cost. Each agent visit uses bandwidth, compute, and sometimes a paid API call. A site protecting its servers from a flood of requests has an incentive to refuse them.
2. Content theft. Publishers don't want their articles scraped and fed to a model they don't get paid for. It's why many now block AI crawlers in robots.txt.
3. Metered access. Some sites are selling agent access. Cloudflare rolled out "pay-per-crawl" billing that returns an HTTP 402 payment-required status when a crawler like GPTBot has no paid arrangement. So "blocked" is sometimes really "pay up."
4. Headless browsers. A datacenter IP doing hundreds of pages a minute is an obvious bot signal to detectors like Akamai, Cloudflare, and Datadome.

What actually works

There's no single magic bullet, but the fixes fall into a clear order. Headless browsers and datacenter IPs get blocked first, so start from the weakest link:

1. Use the user's real logged-in session. The most reliable fix is to have the agent drive a browser the person is already signed into, not a fresh headless session. Session cookies and a history of behavior are far harder to flag than a cold fingerprint. This wins for paywalled and logged-in content, which is exactly the hard part.
2. Move to residential IPs. A datacenter IP screams "bot." Residential or mobile IPs are how legitimate public-data collection survives, and self-hosted stacks often combine them with a proxy.
3. Blend in rather than hammer. Slow down request rate, respect robots.txt where you're allowed to, and avoid the tell-tale burst pattern of dozens of pages a minute.
4. Read the 402. If a site returns payment-required, it's not a vulnerability to exploit — it's a business model. Either pay for the allowed quota or move on.
5. Add a fallback for the sites that matter. Keep a short list of the few pages a person actually needs, and give the agent a route through when the default path is walled off.

The most common mistake is treating this as a scraping problem and reaching for a sharper headless browser. That usually makes it worse: a more aggressive fingerprint is what the detection side is tuned to catch. The better mental model is access, not evasion — you're trying to get the same content a person would legally get, using the same identity and behavior. That's why a real signed-in session keeps winning where a proxy alone gets blocked again the next day.

The takeaway

Blocking is a solved problem for the right reason: the most reliable way to get an agent onto the open web is to borrow a person's real, signed-in session rather than send a headless browser from a datacenter. Bots outnumber humans on a big slice of the web now, and every extra layer of detection is a reason to use a session that looks human instead of fighting the detectors head-on. If you're building an agent that needs the web, solve web access before you solve prompting. For how to wire that up safely, the agent permissions guide is the place to start, and wolffi.sh/start covers the setup end to end.