Rate limits vs. bot detection: why throttling scrapers is not enough
Short answer: Rate limits cap how fast one client can pull pages, but scrapers do not need speed. A bot pulling ten listings a minute from a thousand rotating IPs stays under every threshold and still walks off with your whole inventory by morning. Stopping scrapers takes detection: knowing which clients are automated, whatever rate they run at.
What rate limiting is good at
Rate limits protect infrastructure: they stop one client from hammering your servers and make denial-of-service attempts expensive. Every marketplace needs them. The mistake is treating them as a security control against scraping, which is a different problem.
Why scrapers do not care about your thresholds
- They parallelize. A scraper limited to 60 requests a minute per IP uses 100 cheap proxy IPs and gets 6,000 requests a minute.
- They cache and diff. Smart scrapers fetch each listing once and only re-check for changes, staying far below any threshold you could set without hurting real users.
The threshold trap
When throttling fails, the instinct is to lower the limits. Tighter thresholds catch real users: shoppers comparing listings, dealer tools syncing inventory, your own app polling for updates. The scrapers just add more IPs. You cannot win a rate war against an adversary whose marginal cost per IP is near zero.
What detection adds that throttling cannot
- Intent over velocity. Detection looks at what the client does, not how fast: systematic traversal, form-field probing, inhuman session patterns. A slow scraper still behaves like a scraper.
- Identity that survives rotation. Browser fingerprints and automation-framework traces identify the tool behind the request. Block the tool, not the IP.
- Graduated responses. Route suspected scrapers to degraded content: stale prices or watermarked images. The scraper's data becomes worthless while real users never notice.
- Coverage of the quiet ones. The most damaging scrapers are the polite ones your rate limiter never sees. Detection is the only layer that finds them.
The right stack for a marketplace
- Keep rate limits as infrastructure protection, set generously enough that no legitimate user ever hits them.
- Add behavioral bot detection as the security layer, tuned for your vertical's scraping patterns: inventory harvesting, VIN collection, price monitoring.
- Feed confirmed scraper identities into your edge rules so repeat offenders get blocked before costing you another request.
- Measure listings scraped per day and fake leads blocked, not requests throttled. Those numbers tell you if you are winning.