Options discussed on the Fedora Infrastructure weekly meeting:

• ipset in ansible

Let's look at difficult it will be to have the ipset of blocked IPs in ansible (either private or public) which makes it easier to collaborate on it and comment with reason for blocking them

• Deploying anubis in Fedora Infrastructure

Anubis is a service that is trying to block AI scrapers. We can try to deploy that in Fedora Infrastructure and see how it works.

• mod_qos for httpd

This will not solve the issue with AI scrapers, but could help to make as prioritize our own IPs so our services are still be able to work even when we are under attack.

@james mentioned in the standup later about possibly using nftables rate limiting... thats another option.

As the AI scrapers are not using Firefox user-agent anymore, I don't think the anubis will work anymore.

FWIW, I have been able to mitigate things by 403'ing the specific projects they are hitting as so far those have all been old mirrors with no changes in many years.
Someday soon tho, they might hit something active...

I blocked a lot of chinese IP's today on pagure.io. Added them to spammers ipset, but there wasn't a rule for the ipset, so I added one.

It helped, I just hope I didn't do anything wrong.

Maybe another tool to look at when dealing with AI scrapers https://zadzmo.org/code/nepenthes/

I've found looking at the projects they are hitting hard and 403'ing them in apache is much more effective than trying to block ip's. There are just too many. ;(

Note that we moved things to nftables today, so blocking ip's will change a bit...

Another tool that was recommended on matrix for rate limiting API https://github.com/apache/apisix

Another one that could help shared by @arrfab https://github.com/ai-robots-txt/ai.robots.txt

I'll mention that we could also use amazon cloudfront (It has a bot protection thing).

@kevin That would work only for AWS hosted machines, yes?

Nope, it could work for any machines. It would need changing dns to a cloudfront cname, but then users would hit cloudfront first and it would cache things from the origin.

I'm not sure how well it would work for complex apps, and using a non free service isn't something I like, but it is an option.

I had really nice discussion with OpenSUSE folks on DevConf.CZ. They are dealing with scrapers using berghain, which is haproxy plugin. There is more about it in this blog post.

I was surprised how similar their infrastructure is (haproxy, mailman, pagure).

Another one shared on infra matrix channel https://cgit.rory.gay/BlockScrapersC.git/

Another one (and links to others off it):

https://git.gammaspectra.live/git/go-away

I've setup anubis on stg.pagure.io now. (need to still land the changes in ansible).

Seems to be working nicely.

Closing this and tracking deployment in https://pagure.io/fedora-infrastructure/issue/12784