1
Rate limiting is a roadblock, not a speed limit (opens on the source site)
Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.
#access-handling
8 posts · 1 company · newest first
From the web
2
CAPTCHA is more than just a puzzle piece (opens on the source site)
The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.
3
Inside antibot, the arms race you can't see (opens on the source site)
Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.
4
Firewalls run the web, just not on purpose (opens on the source site)
Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.
5
Four in 10 sites block AI bots with robots.txt (opens on the source site)
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
6
75% of the web uses robots.txt - here's how (opens on the source site)
Three in four of the world's top sites publish a robots.txt, yet few name individual crawlers. A look inside the web's advisory layer: coverage, sophistication and limits.
7
Revealed: How retailers use tech to block the bots (opens on the source site)
The web’s big general marketplaces exist to be browsed by everyone. But they are also the second most defended sector online, shutting the door on AI crawlers.
access-handlingexcerpt only · body stays at the source
8
Fashion websites are the hardest to size up (opens on the source site)
While publishers fight over AI crawlers, fashion has built some of the most heavily defended shop windows on the web. So, which controls are in fashion, in fashion?
access-handlingexcerpt only · body stays at the source
8 shown