r/technology • u/lurker_bee • 5h ago
Artificial Intelligence Websites are closing their doors to AI agents
https://www.theverge.com/ai-artificial-intelligence/1013195/ai-agents-blocked-web70
u/WhichCarry749 3h ago
Half the time Cloudflare blocks me for just opening three tabs too fast, but the bots scrape fine.
511
u/__OneLove__ 5h ago
Techbros believe they should have the ability to access the Internet and steal anything and everything, so that they can turn around and charge us for the same.
🤦🏻♂️
202
-8
u/flockbush 1h ago
You’re getting training and inference mixed up. These are agents just querying websites to do tasks for people, not scraping them for training data.
-30
u/ulimn 4h ago
You are totally right, but I think it has another side as well. Normal people not only pay the price of their content being stolen, but also when you and I try to ask chatgpt (or alternative) to search for information, it will be blocked.
The problem is that the block should be there for training but not for normal usage (inference) imo. But afaik it’s not possible to do.
17
u/Kwabi 4h ago
That still kills the incentive structures of the internet.
The only websites that will be left are people who want to manipulate AI, people who want to get the AI to sell something to you and good samaritans (who pay for creating and hosting high quality information completely out of pocket despite knowing that their content will be taken and their name will be stripped from it).
You don't want a world where the internet is only useful if good people are willing to hurt themselves for nothing.
-7
u/ulimn 2h ago
What you just described is what was basically SEO for the last 20 or so years - and I agree it sucks.
And please try to read again what I said and try not to fill the holes with your imagination. Especially don't try to put words into my mouth.
Also, I would suggest you educate yourself on the topic.
8
43
u/QuimGracado9 3h ago
I help run a small oldschool forum that has maybe 30 active members at a time but for the last 18 months it's being constantly DDOSed by AI crawlers. The record of most users online increased by 1000 people because of them
21
u/WhollyHolyHoley 2h ago
I run a few small websites. I have had them for 10+ years.
In the past year I have had to implement crazy security rules because I keep getting swarmed by bots, sometimes 200 thousand or more pings in a day. The server starts delivering 429 errors. It is so frustrating.This is why we can’t have nice things.
20
u/B00marangTrotter 3h ago
I'm in the same boat, my groups old school forum has been flooded. It's been hosted by godaddy for decades, and we were about to leave but they partnered with cloudfare back in April and released tools to block and monitor ai bots and web crawling.
Since we blocked everything we're back to normal.
1
u/TheGloomyWoodpecker 12m ago
Respectfully, you should find anyone but GoDaddy. You’ll save money too.
101
u/tallandgodless 4h ago
Why won't they let the theft machine do theft on them? So selfish!
-36
u/FrickinLazerBeams 4h ago
That's not what this is.
15
u/tallandgodless 3h ago
How is running scrapers on every site to see what's no nailed down not stealing?
0
u/liright 2h ago
AI agents aren't for scraping content. AI companies use completely different and much faster tools for that. AI agents are doing the work of humans, like when I tell Claude to find me a specific case for my iPhone that fits criterias A, B and C and it goes searching through the entire internet to give me results.
6
u/creepy_pancake 2h ago
You are forgetting a very important detail, which is ads. The good majority of the websites that provide the data you describe rely on ads for revenue. Guess what happens when AI just scrapes the data.
0
u/FrickinLazerBeams 2h ago
We're not talking about scraping content. That's what happens for training AI models, and yes, it's a real issue that we should be critical of.
This conversation is about agents using a website.
I'm not saying we shouldn't also be critical of that but we should at least understand what we're being critical of, or we're just screeching nonsense.
0
13
11
142
u/sysadmin_diaries 5h ago
the real problem is how websites can tell the difference between an ai agent working for a real customer and a bot trying to scrape data or abuse their services. blocking everything might be easier for security teams but it also makes these agents pretty useless for normal users. i think trusted api access with proper permissions would make more sense than having agents pretend to be regular browser users.
95
u/CakeTown 5h ago
But at the same time not all webmasters will have the patience or bandwidth (literal or metaphorical) to support a robust API and Id fully support any maintainer who just wants to shut that door. ‘Legitimate agent’ or not, if it’s an automated system then it’s likely to draw more resources and require more safeguards than your average user putzing around a website.
35
u/PipsqueakPilot 4h ago
Also, you can’t advertise to an AI agent. Requiring customers to manually use your website puts eyeballs on ads.
32
u/Trump_Russia 4h ago
oh man i left a big tech company a year back and their executives are 100% convinced AI agents are just going to shop for everyone...
Like they were honestly hoping the industry would towards AI ordering a product you didn't even know you needed...
I remember an someone being like "Let's say Jim schedules a trip to Denver for a hike with friends, AI should know that Jim doesn't own a pair of hiking shoes and was telling a friend he needs to buy them for the hike... and proactively buy the right size and fit without Jim needing to."
And my jaw fuckin dropped...
And multi coworkers immediately slacked me like 'the fuck is this dystopian shit??"
They really want to build a world where AI makes decisions on how to consume for you.. and automates the entire process. It's fucking wild.
10
u/TheG00dF1ght 3h ago
I will never, ever, allow anything to shop for me. Fuck I won't even use amazon because I still prefer to touch and feel something before I buy it.
1
u/moonski 11m ago
for real like, AI agents are very fucking useful for quite a lot of mundane bullshit... but letting it use your credit card (or debit card whatever) just seems absolutely fucking insane to me.
The furthest I'd ever go is letting it generate a shopping cart or list for me but never, ever giving it any way of actually paying
1
u/Throwawayhrjrbdh 1h ago
Realistically this is going to happen but there is just going to be a prompt asking if you want buy. We are already sorta there with amazons rec algorithms and such.
The main thing is some just won’t have the money to not care what it buys and even those who do typically might not fancy a 100k grand piano showing up one day alongside a bunch of movers
7
u/Version_Sensitive 4h ago
And people with at least 1% of tech knowledge already use ADP or ublock because absolutely everyone hate invasive ada
1
u/redaemon 2h ago
I'm just waiting for the unrelenting wave of AI powered attacks against IoT devices.
Maybe we can finally go back to dumb devices lol
-6
u/ICantBelieveItsNotEC 4h ago
It'll be the other way round in a few years. When most of their users interact with their site via agents, webmasters will be questioning whether there's any point maintaining a human-readable site for the handful of boomers who still use the legacy web.
5
3
23
u/X4roth 3h ago
Websites/services are under absolutely no obligation to provide API access or otherwise accommodate AI agents or other automated users. AI agents aren’t “normal users” and you can’t make the assumption that an AI agent corresponds 1:1 to a real human being. You can deploy a fleet of agents. A huge fleet. AI companies are essentially providing general access to a button that says “make a bot”. Bots can make bots. This very quickly becomes overwhelming and drowns out access to actual human users.
6
u/ItsSadTimes 2h ago
Unless they make a separate API endpoint that charges the agent per request, they're under no obligation to make it easier for bots. Even the companies selling the AIs and agents know fully well that this business model is unmanageable and will eventually eat itself by driving out all the sites that actually provide the information they use. So, to secure those sites, fuck them agents.
21
u/RomanCow 4h ago
> but it also makes these agents pretty useless for normal users
Why should they care if it makes some other corporation's product that they make no money from useless? Even "an ai agent working for a real customer" likely means they still have to pay the same server costs without the actual human eyeballs on their site, so an overall loss for them.
-1
u/liright 3h ago
Because if I have my AI agent search through the internet for a specific desk chair that fits my criteria and one website blocks my AI agent, then they've just lost a potential customer.
3
u/Sweaty_Mushroom5830 2h ago
Amazon already lets you do that by filtering price, weight,lumbar support among other parameters, and by the way so does eBay, Wayfair, IKEA and many more so your argument is pure either laziness or incomplete
1
7
u/not_right 2h ago
You need an AI agent to do an internet search? Maybe you could ask it to tie your shoelaces for you while it's at it.
-4
u/liright 2h ago
Yeah sure, let me just manually go through 100 different chairs on different websites and compare their specifications and prices. I have nothing better to do than to spend 3 hours on that instead of AI doing it in 2 minutes.
2
u/JackReacher3108 2h ago
Hey it’s not our fault you are too lazy to do your own researching and instead resorting to ai
1
u/not_right 1h ago
Ok serious question though, because shopping online can be a real pain. What criteria can you have an agent search for that is better than just google? I find one of the biggest issues with online shopping is specifications not being listed. In your example of a desk chair what kind of specifications would someone put in?
1
u/liright 35m ago
If specs are not listed, the AI automatically searches other sites that have that product until it finds one where they are listed, then it shows me that one. If they're nowhere then it just ignores that result. Then it just gives me the few results that it found that fit my criteria and I can pick from 2-3 options.
In your example of a desk chair what kind of specifications would someone put in?
With a desk chair, I would personally look for one that has adjustable height and lumbar support, ability to block the wheels, has ventilated back, is in black color and costs under $350. And obviously good reviews. AI can sort through hundreds of options and selections and show exactly the few I that fit what I need.
1
u/Sweaty_Mushroom5830 1h ago
Go out touch some grass my friend, because when I was looking for chairs I didn't have to ask a bot I simply went looking for what I could afford, what fit me and I found it in less than five minutes and I didn't use bots
-9
u/ICantBelieveItsNotEC 4h ago
Because their website will be irrelevant in a few years when agents are the primary way that most people interact with the web.
7
3
u/RomanCow 3h ago
Maybe. But they probably don't think we are at the point yet where that's the case -- i.e. at the present moment, they make less money letting the agents in. Their sole concern is reporting more profits to their shareholders this quarter than last quarter. Anything else is mostly meaningless if it doesn't serve that goal. And if enough of them doing this does in fact "make these agents pretty useless for normal users", then it will be a long time before it reaches that point, which they are more than fine with.
6
u/nickmullen_real 4h ago
every time I tried to search on amazon today, it asked me to check a box to prove I'm not a bot and sent me right back to the home screen. guess ill buy it somewhere else
2
48
u/Nosiege 5h ago
Why should websites care that agents are useless for normal users? Agentic workflow is a mistake. Just like 3d TV and hd DVD
-18
u/1_________________11 4h ago
Idk how you can say that ive had my agents do some pretty helpful things for me from deploying software to troubleshooting media files that weren't playing properly. To reverse engineering stuff while I slept. Also evaluating my finances and retirement composition and planning out financial instrument purchases.
I self host both the agent and inference box
13
u/Nosiege 4h ago
You missed the point.
-10
u/1_________________11 4h ago
Agentic workflow works buddy think your missing it.
11
u/Nosiege 4h ago
My point was why should a website care if a regular users agent doesn't work.
-6
u/1_________________11 4h ago
Might move to a service that allows my agent to work on it. Usually websites want visitors and users to use their service if they make it difficult im sure others will replace them.
75
u/DinobotsGacha 5h ago
AI is pretty useless for normal users.
-11
u/1_________________11 4h ago
I was having my local agent try to use a few sites just to grab my own data and the damn site kept blocking it ended up grabbing the data for it another way. But pretty frustrating.
4
u/ptear 4h ago
What do you use your agents to browse the web for generally?
-6
u/1_________________11 4h ago
I was having it grab data from a app i pay for and use. But besides that research reading docs so it stops making dumb mistakes hah. One found some local places to buy some things from for some gifts was pretty cool since i was able to support local businesses that way and didnt need to go outa my way to research which one had what i needed.
-17
u/eras 4h ago
Not long ago I used AI to find a carrying bag for Novation Launchkey 37 MK4 from Thomann, which only has dimension as a search criteria for certain categories, and not all bags are in those categories.
It found a perfect match. It would be quite annoying to manually traverse it through.
It was also able to find restaurants in a destination city that had dairy-free items in their menu. I can't of course be sure if those were the only places, but it was also easy to verify if its findings were correct.
Both seem rather normal—and useful—use cases to me.
Granted it's pretty useless if you have no imagination.
13
u/schooli00 4h ago
It's the other way around for me. Using AI to do product research is completely useless. It confuses facts between similar products so much. Confuses interior exterior packaging dimensions just to tell you everything fits or matches. It's overly confident and overly agreeable like a slimy used car salesman.
8
u/DinobotsGacha 4h ago
AI use is growing, sure. Your use cases were served just fine by google searches before AI.
-10
u/eras 4h ago
Really? Google has the capability to find restaurants with menus supporting certain allergies? Or finding number ranges describing object internal dimensions, according to the search?
I must have been misunderstood Google's capabilities.
10
u/DinobotsGacha 4h ago
Yes, I have been using Google to find kids menus, vegetarian items, and happy hours for years. AI is trained on the same data Google has been web crawling for decades
-6
u/eras 3h ago
It's another thing to crawl the data—and AI is using search anyway, so the results of the crawling—and another to use that crawled data and evaluate it under arbitrary criteria, and report it in desired form.
For example, OpenAI was able to find vegan restaurants at a destination and put them on the map, along with distance from lodging to said restaurants. (Granted I used MCP for the map part, but I think it is able to do that by itself nowadays.)
0
u/DinobotsGacha 3h ago
Evaluating data against criteria is what searching does and google maps does the other stuff. However, its great you're diving into AI and liking it. This tech is still maturing and one day we may all have AI agents acting as our personal assistants OR overlords.
4
8
u/UntowardHatter 4h ago
Sir, you've discovered Google.
-4
u/eras 4h ago
Just as a benchmark, can you find that carrying bag for Novation Launchkey 37 MK4 from Thomann with Google? We can then compare which internal dimensions hit closer.
12
u/UntowardHatter 4h ago edited 4h ago
Took me like 10 seconds.
https://www.thomannmusic.no/gator_gk2110_multieffect_bag.htm
This one also fits.
https://novationmusic.com/products/25-key-keyboard-backpack
This one is also great. Keyboardist in my band uses it:
https://sequenzmusicgear.com/en/products/lineup2/#mp-large-msg
I found these with Duck Duck Go, with AI disabled.
AI basically just scrapes Reddit anyways.
-2
u/eras 3h ago
Well that was pretty good! But it does not say anything about how soft its insides are (judging from dimensions, not very), and no pictures from the inside either.
AI's find was https://www.thomann.fi/flyht_pro_hsc_hard_shell_case_s.htm which is a bit thinner from the inside and says to have foam inserts.
7
13
u/calidriel 4h ago
good they should be useless for normal users
-3
u/liright 3h ago
"Grug hate fire. Fire bad. Fire hurt. No man should use fire."
6
u/calidriel 2h ago
fire was a useful technology and not a hallucination machine propped up by the worst people on earth
0
8
2
u/ih4teme 4h ago
Behaviors allow them to know if its AI or a person. There are little markers on a website to tag mouse movements and other things that confirm human use. AI moves in different ways that don’t follow human behaviors.
With the advertising component, they want humans so they can validate it was a human hit from an impression standpoint.
I think main issue is still DDOS when it comes to AIs overloading sites. Hence the concern with replication.
4
u/Front_Caramel_1723 3h ago
Thats not the question. The question is if they want to block AI agents that work for a real customer, i.e. a customer telling a bot to search/buy the best rated couch with dimensions xyz.
3
u/No_Ocelot_2285 39m ago
This is useless for most web sites. It can’t prevent scraping or bot stampedes. Once a web page is at the point where you can measure mouse movement, it’s too late - the page and all of its content had already been requested and rendered and delivered to the remote browser/bot. Your content has already been stolen, your server resources have already been wasted, and the bot on the other end can just create a new browser/session on a different IP for the next request.
2
2
u/Vegetable_Tension985 52m ago
you're dreaming. completely unrealistic. The AIs gonna be doing whatever they want because it's a hard problem to stop and it will become an impossible problem to stop
2
u/mattcannon2 4h ago
Why should a news website care about an AI agent other than one that the news website has pre-agreed with the owner?
1
1
0
u/ryuzaki49 4h ago
Captchas in 2027:
Please send a handwritten letter to our PO box to probe you're human. You will be able to access our site your answer in 3 weeks
0
u/wachuwamekil 3h ago
I’ve had mine using a vm, and interacting with websites via connected hardware mostly. If there is a Captia it asks me for help. It takes screen shots and ocr’s them. Much slower but zero blocks so far.
-4
19
u/Captain_Tooth 4h ago
A bot time. If a person needs an AI agent to shop, we are in a low part in society.
13
u/defenestrate_urself 4h ago
I wonder how much AI's like the one embedded into Google search is diverting ad revenue from websites.
Often you don't need to visit a website anymore to access the information, I'm guessing it's a huge impact.
6
u/FeatherlyFly 4h ago
There have been articles about smaller and niche sites shutting down from a mix of no more revenue plus increased costs because on the back end, LLMs are accessing the site in ways that cost the host more than the usual human ways. That last is IT stuff, I don't understand the details.
3
u/JesT_Weird 2h ago
This is the inevitable outcome of humans treating "clicks" like they were commitments from real people about real things that mattered.
Business has been confusing "activity" for "intention" for decades now, like uninformed idiots.
2
8
u/IntelArtiGen 5h ago
They're also closing their doors to humans they misidentify as AI agents, with a lot of false positive now that many agents are capable to control computers like humans.
For all these websites where people use an agent to pay for a service or buy a product, I however don't see why these companies would complain. Maybe if there are scalpers (in which case they can raise the price) but otherwise what's the problem? People paid.
7
u/moreobviousthings 5h ago
You hope people paid and intentionally so. But how to verify that an actual person intended for the agent to commit to payment?
-5
u/IntelArtiGen 4h ago
Why would I care? If someone made a mistake and spent money on my store it's their problem. If they want a refund, I substract the cost of the refund process from what I reimburse. I'm sure many stores already have a way to handle abuses like this. Maybe if you do 1 mistake it's fine, but if you massively use agents wrongly to spent money, well that's your money and your problem.
13
u/BrainOnBlue 5h ago
Part of it is that it's a customer support nightmare.
Agents are going to make mistakes (or people are going to give their agents bad instructions, but you have to deal with that either way). The number of extra customer service reps you need to handle the increased number of returns and such might not be worth the extra sales you might get.
And yeah, sure, you could have AI handle the returns, but at least in my personal experience with Amazon's customer service AI bot, those still have to drop you to a human pretty frequently.
-9
u/IntelArtiGen 5h ago
I'm sure they'll make mistakes but can't they just make people pay for the mistake? I'm sure there are ways to put that cost on customers. If humans make less mistakes you just put a threshold saying everyone can make 1 mistake per year but otherwise you'll pay if you want to send back something (you won't be fully reimbursed).
2
u/BrainOnBlue 4h ago
Sounds like a great way to piss off the AI agent-using customers you just changed your policies to accomodate.
1
u/FeatherlyFly 4h ago
Maybe. Having different policies for different customers based on how you order will piss people off.
Having a warning on your website won't help because the pissed off people won't be on your site in the first place and the bots aren't sentient to understand warnings.
So you'll end with pissed off customers who won't order again and may review bomb or stress put your customer service employees.
For lots of businesses, especially small ones, the lemon ain't worth the squeeze.
2
u/Ok-Trainer3150 3h ago
As if that's even possible. And I'd wager that if there's money to be made by cutting costs or increasing traffic, they're all in. It's very apparent that the internet is increasingly been driven by AI.
2
u/CautionarySnail 57m ago
I’m for this.
If the sites wish to permit agent access under their terms, they can build an MCP server and provide access that way. That’s what MCP is partially for, to create uniform safe paths that limit the damage when hallucinations occur by restraining the agent’s actions to a limited subset.
-4
u/brandibesher 4h ago
Claude couldn’t access Amazon when i asked it to check payment methods on some orders.
-1
u/EndogenousBacon 4h ago
That's interesting, this explain why I end up buying things on lesser known websites bc it's not giving me the results from Amazon
-3
u/AlpineAvalanche 4h ago
How do they plan to keep them out as they get better at hacking and pretending to be human?
7
u/DopamineSavant 4h ago edited 4h ago
There is a difference between being allowed to do something and hacking in order to do something you've been prohibited from doing.
0
u/plsobeytrafficlights 2h ago
ai agents are hacking and stealing crypto wallets and people think they can keep them from reading your web page? goodluck with that.
-1
u/W1v2u3q4e5 4h ago
How are AI agents actually being blocked by websites though? Aren't they supposed to simulate natural human ways of accessing websites as smoothly/progressively as possible? Quite interested to know about this and of course there are browser fingerprinting techniques, captchas, rate limits and so on, how AI agents in particular are blocked or at least have their impacts reduced is necessary to know for websites that need to actually avoid them.
4
u/Splinezzz 4h ago
It depends if they self identify correctly.
For instance if they identify as a bot in like a user-agent header or similar. I.e like AppleBot or similar.
If they identify as a normal user on say windows/chrome and behave in a human like way it is very difficult to tell.
3
u/RyanCargan 4h ago
Part of me wonders if people will start doing weirder stuff like HTML rendering inside the canvas, adversarial rendering within that canvas so that even costlier vision models won't be able to grok it reliably, using obfuscated WebAssembly to store most of the site code to make it costlier to parse if parsed at all, etc.
Would break the semantic web ideal, which affects both beneficial crawlers and vision-impaired people using assistive tech, but I wouldn't put it past some folks to eat that loss.
It's kind of a weird middle ground between auth-guarded non-public sites, and standard public sites.
The main point is basically making it costly to scrape by machine if anyone ever bothers.
I don't think most would discard the accessibility, SEO, archiving, and other benefits just to get one over on the bots buuuut... nothing is really stopping them in theory.
2
u/Black_Moons 2h ago
I for one can't wait for proof-of-CPU webpages that run the webpage text through a decrpytion algo 10,000 times so that it takes my PC its entire CPU 10 seconds to decode the text.
Annoying for humans, devastating for anything trying to crawl thousand of webpages every second.
1
u/RyanCargan 2h ago
Would be funny yeah, but the above tricks don't add any friction for human users or their machines, except in the accessibility case... and then there's the SEO sudoku...
-7
u/corobo 5h ago
If it were the crawlers that inhale content for training, aye sure I get it, but this is basically "websites block specific browser" - go for it, you just won't be in the results anymore.
If that's what you want, sweet. If you're a competitor of something I'm looking for, you're not anymore.
-3
-2
u/bitspace 2h ago
These are the ones that are going to miss the bus. I'm not sure what Amazon has up its sleeve, and I certainly don't count them out, but the writing is on the wall for consumer capture.
800
u/fuzzus628 4h ago
Why did we all do captchas for the past 30 years only to let the robots in now?!