Nginx has an assortment of options to rate limit people/bots. [1] For the 10% not using a user-agent generic rate limits could be applied based on volume limit_rate_after. One would have to get creative with nginx maps. Another option would be to calculate patterns in your access logs and just blackhole or ipset reject IP addresses or networks that are abusive accepting some of them may be abusive humans vs abusive bots.
I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?
I mean, not sending 418 and sending 429 or 403 or anything more useful would be on my list of things to try. Might double check what 418 is while you're trying to figure out why sending that doesn't seem to do anything.
Nginx has an assortment of options to rate limit people/bots. [1] For the 10% not using a user-agent generic rate limits could be applied based on volume limit_rate_after. One would have to get creative with nginx maps. Another option would be to calculate patterns in your access logs and just blackhole or ipset reject IP addresses or networks that are abusive accepting some of them may be abusive humans vs abusive bots.
[1] - https://serverfault.com/questions/639671/nginx-how-to-limit-...
I feel for you. Same for OpenAI (https://www.linkedin.com/posts/appinv_openai-is-a-felon-comp...). A friend told me Ai bots even ignored the robots.tx ...
I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?
> What else do I need to do?
I mean, not sending 418 and sending 429 or 403 or anything more useful would be on my list of things to try. Might double check what 418 is while you're trying to figure out why sending that doesn't seem to do anything.
Changing the status code didn't help. I eventually just withdrew the server from the internet.