r/Library • u/kped320 • 16d ago
Local Library Bot searches
Hello everyone!
I work in IT department at a library and we're seeing an increasing number of bot searches and bot traffic. Anyone else experiencing this been hearing about this?
12
u/Otterfan 16d ago
Good lord yes.
I run a number of digital library projects, and a couple of them are getting .5 million+ attempted bot visits a day. Dealing with them has become close to a full time job.
2
u/kped320 16d ago
What have you tried?
3
u/Otterfan 15d ago
The best solution is Cloudflare, but for organizational reasons, that isn't really an option for us (yet).
All of our requests are proxied through nginx, and we've gotten very good at blocking them there. We do two things:
- Block IPs that are obviously bots. Unfortunately, 99% of our bad requests come through residential proxies, so we need to block hundreds of thousands of IPs for this to work.
- Block requests that match bad patterns. Most of the bot requests are complicated (lots of faceting) with questionable referrers. We set up nginx rules to drop these requests (return 444 codes).
We're looking at setting up Anubis to block bots. We need institutional approval to set up an Anubis instance, but it has a lot of promise.
There's a Code4Lib Slack channel about dealing with bots. It's a good place to start.
2
u/kped320 16d ago
An interesting idea! I like that solution and is elegant! It's insane we're getting 80k hits in 15 minute periods and I can't understand why glad other libraries are having this issue that makes me feel a little better 😂🤣
2
u/midnitelibrary 16d ago
You have to directly reply to comments for people to get notified of your response.
2
u/ContraryMarie 15d ago
C4L Slack is a great place to start. We're slammed regularly. Anubis is what we're testing atm; CF before that.
Lyrasis also a good resource: Aggressive AI Harvesting of Digital Resources; and the WG's Solutions Showcase series
1
u/Rupertcandance2 16d ago
Bywater told us it was happening on the Aspen overlay. Causing so many problems, unhappy patrons. 😞 don't know about anything local though. I rarely see our IT person. He works from home a lot.
1
u/polygraph-net 15d ago
Hi u/kped320
I work for a leading bot detection company. I'm a big supporter of libraries. We'd be happy to provide free bot protection for your library. Please DM me.
2
u/ContraryMarie 14d ago edited 14d ago
What a great use of a library's limited, if not nearly non-existent, funding.
[edit] "Free" is never truly free. This "free" protection may require data sharing, and certainly would require resources on the side of the library and/or on the side of the poster's enterprise (remunerated, ofc).
The poster provides neither name nor link of their service, just a vague description and demand for a DM. Shady.
0
14
u/midnitelibrary 16d ago
Yes. Some libraries are locking down their catalogues and other systems so that you have to login to an account to see results due to the overwhelming number of bot crawlers.