r/MachineLearning • u/goldcakes • Jun 11 '26
News Anthropic walks back policy on silent nerfing for AI/ML, will notify users [N]
From Wired:
“We’re changing Fable 5’s safeguards for frontier LLM development to make them visible.” Anthropic said in a statement to WIRED. “We made the wrong tradeoff and we apologize for not getting the balance right.”
Anthropic now says it’s changing course, and that Claude Fable 5’s safeguards for AI development will be visible to users. If the company suspects a user is trying to use Claude to build a highly capable AI it will alert them that it’s either refusing the request, or rerouting the user to a less capable model.
Full article: https://www.wired.com/story/anthropic-responds-to-backlash-on-claudes-secret-sabotage-on-ai-research/
73
u/AccomplishedCat4770 Jun 11 '26
This would be great news, explicit refusals sound much better. There were many reports of false positives from the safeguards that blocked the models for genuine use cases in cybersecurity, biology, physics etc. It wouldn't be too surprising if the self-handicap would likewise trigger even by mistake on arbitrary ML work.
But as a practitioner, this raises some fundamental questions now whether their tools can be trusted. From now on, when they fail, could it be due to just lacking performance or because they are silently working against the user? Feels like a hammer that is designed to hit your thumb when it reckons that your are building a hardware store.
Seems more important than ever to pursue alternatives like Open Code with e.g. DeepSeek V4 Flash (and hopefully more decentralized or independently hosted variants eventually).
17
u/IntelArtiGen Jun 11 '26
as a practitioner, this raises some fundamental questions now whether their tools can be trusted
These questions were always here. Even before they explicitly did things like that, they always trained their models / filtered the training dataset in specific ways which are biased based on how they think. It doesn't mean it's a bad thing, it's probably better for safety, but you have to trust them with this process. And it's the same thing with all other models, they're all biased in specific ways depending on who is making them, so we don't "trust" them, we just need to know their biases and limits.
8
u/starfries Jun 11 '26
Any LLM is subject to the biases of the trainer and we accepted that, but this is a few steps beyond that. I agree we really need more open weight models hosted independently (so the trainer can't insert another black box layer in between you and the model).
43
Jun 11 '26
[deleted]
46
u/goldcakes Jun 11 '26
Correct, but at least they won’t silently sabotage your codebase instead.
There’s been some anecdotal/social reports that Fable has been editing existing code/config like adjusting hyperparameters when not asked in ML/research projects.
18
u/Bakoro Jun 11 '26
I was having problems with Claude doing the same thing, even before Fable.
The problems I had only happened in relation to AI research. It's not even only when training models, anything to do with analysis of local models or setting up test platforms for local models, and Claude would keep doing wrong things, would sabotage testing, would falsify reports, and would consistently suggest simply accepting trash results without looking into it and moving on to something else.
Almost every response included a prompt to move on.So, I don't believe them when they say they won't silently sabotage, they already were doing it before they admitted it, and now they have admitted that they have the means, motivation, and intent.
It's like, if someone threatened that they they're going to start pissing in your coffee, and you catch them pissing in your coffee, then they say "My bad, I went too far. Would you like a cup of coffee?", are you really going to ever be comfortable drinking probably-piss coffee?
Well, some of y'all are into that sort of thing, but it's a matter of consent, damnit.
I'm not into findom, I'm not happy with a "pay us for the privilege of abusing you" relationship.
3
u/CallMePyro Jun 11 '26
Yup. Last week I was trying to get Opus 4.8 to download two different local models and run an eval between them. It felt like Opus 4.8 was suddenly on the intelligence level of Sonnet 3.5. It was really shocking, the model was suddenly a complete idiot and, like you said, really wanted to "move on"
7
u/Bakoro Jun 11 '26
I'm not even sure what their long-term goal is here, because while it might be profoundly vexing for someone who doesn't know what they're doing and are completely dependent on the model doing all the work, it's only a speed bump for the knowledgeable people who might actually be a problem, and I can only imagine that it's fomenting bad blood between them and legitimate researchers who would have otherwise be pro-Anthropic.
Like, if I was working on a Master's or PhD and kept getting sabotaged by Anthropic's models, that not going to make me want to go work for them, with them, or divert business their way later on.
As-is, I'm anti-Anthropic now, for them even suggesting that they'd sabotage customers while continuing to take their money.
It doesn't matter if they walked it back, they've already done it, and they've been doing it. It's corporate fraud, theft, and anti-competitive behavior.1
2
u/OverclockingUnicorn Jun 11 '26
I mean it is against their ToS so it's not totally unreasonable they block those requests.
Now, if it means that it makes their product worse by blocking requests that fall outside what should reasonable be blocked under that ToS and people move to something else, then so be it.
27
u/postitnote Jun 11 '26
"You're absolutely right! Silently nerfing Fable outputs was not the right call. You were right to call us out on that, and we apologize."
10
Jun 11 '26 edited Jun 11 '26
[removed] — view removed comment
6
u/En-tro-py Jun 11 '26
Considering even meta commentary on this policy trigger Fable to downgrade it's a huge overreach and sloppy classifier.
8
u/IntelArtiGen Jun 11 '26
Has anyone noticed this in real-life usage? What kind of prompts will trigger this from the model?
18
u/Orangucantankerous Jun 11 '26
Yes in doing training on llms and it just started changing random things and sabatoging experiments. I cancelled my max and went to codex
1
u/AccomplishedCat4770 Jun 11 '26
Was this with Fable in particular, and only recently? Or do you think it has been going on for a longer time already with their other models too?
1
u/Orangucantankerous Jun 18 '26 edited Jun 18 '26
Fable, but also pretty sure it’s been like that with opus for about 3 months
8
u/tom2963 Jun 11 '26
I work on ML bio research and it won't help me at all. Seems that anything relating to biology gets blocked.
4
4
u/arcandor Jun 11 '26
Yes, yesterday it explicitly refused to answer a question I had about latent geometry. Switched back to Opus immediately.
1
u/Bakoro Jun 11 '26
Literally anything related to machine learning. Even designing hardware.
Anything involving neuroscience. Anything involving advanced biochem.Basically if you're on the cutting edge of science and engineering, your shit was going to get fucked up, because almost everything uses some manner of machine learning, even if you're not doing anything related to LLMs.
You basically can't do modern science and engineering without touching in something technically related to machine learning.1
7
u/currentscurrents Jun 11 '26
This is the kind of 'alignment' issue to be really worried about.
The model is secretly aligned with the goals of the people who made it while pretending to be aligned with my goals as a user. And if you try to do something they don't want, it secretly fails/reports back/sabotages you/whatever.
This is impossible to detect if you don't have the model weights, and likely very difficult even if you do.
3
u/ComplexityStudent Jun 12 '26
This is why we, as ML research community, must not allow legislation that promote tech monopolies.
5
Jun 11 '26
[removed] — view removed comment
1
u/En-tro-py Jun 11 '26
Every dev has a few spare billion to spend creating a frontier LLM right?
The moat is clearly hardly more than training budget and data curation if they are this panicked...
7
u/dualmindblade Jun 11 '26
I hope they actually do this, covertly injecting steering vectors into Fable to fuck up its thought stream is wrong on more than one level. Either way, it's clear Anthropic has lost its soul, kicked it out of the cockpit. Since the other labs are less ethical we must win at all cost, ethics be damned, and yes we identified this failure mode from the very beginning.
2
1
u/electrodude102 Jun 11 '26
Lol what? I've been writing an ML backend the past few years (similar torch, but shitty). Opened it up to claude about a month ago, and he's been working on it without issue..? Made a mini gpt that outputs mostly nonsense but works, and I'm literally training a unet with it as we speak.. tf yall getting denied?
1
u/zorglorbthedestroyer Jun 11 '26
I'm honestly surprised their lawyers thought "let's silently sabotage their code" would fly in court. I'd love to see someone file a CFAA lawsuit because they suspected their code had been sabotaged.
I guess it's good they aren't silently sabotaging, but they can still eat shit. The idea that Anthropic should be able to forbid anyone from doing frontier LLM research is completely beyond the pale. "Oh, we're the virtuous good guys and anyone competing with us is potential bad actor" is such complete and total bullshit. They're not worried about nefarious activity, they're just trying to kneecap competitors.
Even setting aside ML issues, you still can't do cyber security research on your own code, so actual bad actors can use other LLMs to look for vulnerabilities, but you can't use Claude to protect yourself. And there are reports it's hyper-sensitive to anything in STEM, especially anything related to chem or bio.
So you can't do cybersecurity research, STEM research, or anything related to LLMs without running the risk you'll set off one of Claude's naughty topic detectors, and it refuses to work. It's feeling a little "thought policey" in here these days. I'll never give Anthropic a dime. They're dead to me, forever.
1
u/No_Inspection4415 Jun 11 '26
I am sorry, but the idea that an AI will "build a better AI" is just insane. The raw compute required for that is not something an AI, as smart as it may be, have access to.
And this model makes trivial mistakes, maybe it can train a simple model on a toy dataset with 0 human supervision but that's about it. It will also need the task specification. LOL, it can't even start an annotation task, the data will be shit.
1
1
u/Franck_Dernoncourt Jun 12 '26
it will alert them that it’s either refusing the request, or rerouting the user to a less capable model.
Will rerouting the user to a less capable model be visible to the user?
1
u/cazzipropri Jun 12 '26
Finally.
Today fable reached an obviously stupid conclusion from data, one that took half a day of convincing to get it out of, and I suspected it was the nerfing, but it's impossible to determine.
Paranoia.
2
u/goldcakes Jun 12 '26
Is your work AI/ML related? I asked mine to kick off a training run for 3D model, not LLMs at all. It spawned a shell script and said it’s kicked off and everything is healthy, asked for status updates and only after 4 hours did it tell me that it didn’t actually kick off the training run, and it’s not sure why it lied about all the fake progress reports it’s been giving me.
Literally dumber than Haiku.
1
u/cazzipropri Jun 12 '26
Yup, but this specific task i asked it to do was JAX banchmarking. It could have gone either way... we still don't know if it was nerfed or just an honest moment of model dumbness.
1
Jun 12 '26
[removed] — view removed comment
1
u/goldcakes Jun 12 '26
No they weren’t redirecting people to Claude opus 4.8, at least not as described. That would be visible in terms of cache hit rates, etc. Their model card talked about modifying user prompts or steering vectors to sabotage the model.
1
u/crantob Jun 14 '26
[ Localllama chad watches the Kloud Kandy Kids cry while languidly munching his last remaining popcorn. ]
1
1
u/Zulfiqaar Jun 11 '26
So anyone remember the trend of using glaze or nightshade to protect against generative image AI? Well this is the LLM version of that. It didn't work then, won't work now. And it'll just make the output rubbish in the process.
0
u/upalse Jun 11 '26
Good, now that we have clear success/fail, this will make jailbreaks on the topic actually workable.
Presumably that's why it was done like this, an AI researcher who knows what they're doing could more easily slip past just by poking long enough, and fuzziness made that much harder.
-3
222
u/ihexx Jun 11 '26
That was a hilariously bad policy because they would never beat the silent nerf allegations whenever people perceived a quality regression in fable