4 posts
A User-Agent whitelist is not enough to let ClaudeBot or GPTBot through the Nginx Bad Bot Blocker once AbuseIPDB is involved. Here is how to verify AI crawlers against their published IP ranges instead.
Meta's crawlers hit one of my servers at 100 requests per minute from 458 different IPs. Here is how to rate limit the whole meta-external family with plain nginx, without breaking Meta ads landing page verification.
Fake Googlebot traffic can quietly hammer your server. Here is how to verify crawlers against Google and Bing's official IP ranges in nginx and drop the impostors.
The Nginx Bad Bot Blocker script is fantastic and a no-brainer to install on any Nginx server hosting publicly accessible websites. As great as it is though, I’ve recently noticed a significant increase in the number of attacks on the servers which I administer and websites in general. So I looked into into integrating the