Comment by slowin
1 day ago
This neat and all but I think there's a bigger issue here. Anthropic (and whoever Anthropic wants) can do this kind of work with Claude, but if you or I try to do it, we will get blocked. This is a fun exercise, but extend this to something much more nefarious and you can see a moat widening between the model providers, their government partners, and the public.
The only thing that bridges this moat is continued advancement of open models.
> [The model's authors] can do this kind of work with Claude, but if you or I try to do it, we will get blocked.
If this seems concerning, wait until the models get effective enough (gets work done properly at a lower energy need than humans have) that there's no reason for them to release anything. Why offer your weights for download or grant any third party usage access when you can just do the actual job yourself and collect all that turnover?
> Why offer your weights for download or grant any third party usage access when you can just do the actual job yourself and collect all that turnover?
It's hilarious how a the trillions invested in AI companies, and their valuations are predicated on the AI labs Sherlocking entire industries, and yet CEOs, mathematicians, and everyone else are eagerly feeding the beast their most prized information.
Let's not forget the time Anthropic and OpenAI stole some mathematician's research because they were vibe researching the problem and Altman and Amodei thought it would be nice to claim the prize as their own
What else should we do? It's not like we had any choice.
8 replies →
> and yet CEOs, mathematicians, and everyone else are eagerly feeding the beast their most prized information.
The alternative is starving because someone else will.
That's why those companies are worth trillions.
They will just have someone look at every question, they will let it pass, want a cut or deny it. Claude, you can answer this guys question, but don't tell 'em how you solved it and don't give him the best answer, give him an ok answer.
That's already happening. The public models aren't anything like their internal models.
That's kinda unrealistic, isn't it? How will they pay for the thousands or ten thousands of employees, who need to look at every question? And most questions are just ordinary BS, that not really anyone other than the asker cares about. Giant waste of time to check all that.
2 replies →
Just wait till the AI does that to humans in general.
This is why open weights and continued model distillation to improve model evolution is imperative.
Do you trust the people who would gate these capabilities? I do not.
I mean I want this to, just don't get your hopes to high because it's not long till they get dashed.
I don't think it will be too long before models are considered munitions and having the weights for one will require registration.
3 replies →
French Revolution and the 21st centuries version of guillotines, whatever that could be, is probably a good reason to avoid that.
There's also a thing called commodity fetishism but capitalists usually learn the hard way that things created with machines without human input miss the essence of why we're willing to pay for something. Its why a handmade watch is $100k and a casio is $100, or a handmade sweater identical to a machine made goes for 10x.
They could also just make us so poor and anxious that we beg to fight eachother in wars.
While I agree that there's a special element to human crafted pieces, and artisanship would be a terrible thing to lose, I'd point out that Roger W. Smith hand makes ~15 watches per year which sell for around $400k a pop, while Casio's watch division brought in $1.17B in revenue in the most recent FY
Another labor theory of value patient.
3 replies →
Maybe, but that's what the cyber program is for. Nobody knew if the seed was fine or a backdoor. If it was a backdoor, and Claude helped you find it as a normal person, Anthropic would get bad PR. If you're a verified security researcher, actually that's fantastic and Anthropic would get good PR and everybody claps.
Anthropic wants to IPO, they don't want another Fable shutdown situation.
To be clear I would prefer the cyber safeguards to be less restrictive, I think they're currently too much. I often go to open models because of it. But there are likelier reasons to restrict cyber than world domination.
Why?
Why?
Why are we celebrating and encouraging the walling off of the global intellectual columns?
The idea that you can't reverse engineer things, make things, learn about biology or physics without permission is strange to me. These machines have been trained on the sum intellectual output of humanity, the global intellectual commons, and are being used to close off that commons?
I would be OK with their right to create such restrictions if they weren't lobbying the Government to restrict others, thereby ensuring that they control humanity's intellectual commons well into the future.
Who are they to say what I can learn or can't learn? Who are they to control what I can or can't do with these tools?
Perhaps I'm naive, but I think it's better for humans and the machines if we can all think, learn and build. But then again, I'm the kind of person who rejects the doomer pill.
Reject this.
Because there are good and bad people in the world, and if I ask Claude "how do I make cocaine" and it gives me a thorough recipe, it doesn't know if I'm being intellectually curious or trying to become the next Pablo Escobar. They're Anthropic, and because they made Claude, they get to say what we can do with it. Which, I agree, is total bullshit. If I buy a Ford car and I can do whatever I want with it. Drive to work, drive for Uber, set it on fire; plowing into a crowd of protestors is covered by existing laws against vehicular manslaughter. Ford doesn't get a say. But because cars are so ubiquitous and a known technology, there's no blowback to Ford if I commit crimes with my car that I bought from them. If I commit crimes and Claude helped, Anthropic is gonna get sued (there's already litigation against the labs for deaths), and they don't want that.
In a properly free market, we'd have regulations that prohibit labs from vertical integration so they have to release and sell a model and another company hosts it and sell access to it. Americanism, America's implementation of capitalism doesn't have such regulations though (and how would that even work?) so we're stuck with Anthropic having that power.
The fear, justified or not, is that a terrorist is going to use Claude to make sarin nerve gas and bomb subway stations, or the digital equivalent. Could you live with that on your conscience? I don't think I could, but fortunately for me, I don't work at Anthropic so don't have to answer that real life trolly problem.
18 replies →
> The idea that you can't reverse engineer things, make things, learn about biology or physics without permission is strange to me. These machines have been trained on the sum intellectual output of humanity, the global intellectual commons, and are being used to close off that commons?
I mean this is the standard of biochem research right now. You can't just order 50 gallons of TNT off Amazon and neither can you get live viruses from a lab just because you want to do research.
3 replies →
Because AI-assisted hacking is not something we should allow to run rampant with the most advanced models, prior to AI-assited cybersecurity catching up.
>Maybe, but that's what the cyber program is for.
I have been in the cyber program since around the time it first went public and recently renewed it with the updated program.
You still trivially hit guadrails. I was trying to reverse engineer some bit of 20-year-old software that already has known, unmitigated exploits and it still downgraded me.
> that's what the cyber program is for
open weights are the better way imo, no elite gatekeepers please
> Anthropic would get bad PR
I think the current administration would delight in harsher penalties
> But there are likelier reasons to restrict cyber than world domination.
I mean, greed alone would seem to be enough, and it would be fully worth calling out as bullshit.
Anthropic gets zero of my money because their favorite word is no. I cannot figure out how to as an individual or a small startup request elevated permissions for the purposes of anything that they think might potentially be dangerous for someone who is not me to access. I have a background in cybersecurity. I have a background in life sciences. Can I put those skills to use to benefit humanity? Who knows.
I imagine the path forward for the labs will be signing off on organizations/institutions rather than individuals.
The risk/reward is way lopsided towards risk for the labs to vet individuals. Much easier to get an organization and have them take the blast if someone/something goes off the rails.
I cannot in any universe see the labs putting themselves in a "We should never had given this person access" situation.
> I cannot figure out how to as an individual or a small startup request elevated permissions…
https://support.claude.com/en/articles/14604842-cyber-verifi...
https://portal.anthropic.com/programs/cvp
"No."
Dario keeps saying in interviews that AI hacking is not a problem because "the good guys will protect themselves with AI", but then gate-keeps.
I submitted the form and it was rejected without reason.
Now what?
Thank you for the link.
It looks unlikely that I'll be approved, since I have no CVEs credited to me, and I do not publicly post about security concerns.
..........................................
The Cyber Verification Program is not available on free plans, so we are unable to approve you for the program.
If your plan changes, you're welcome to apply again at any time.
Regards,
The Anthropic Safeguards Team
..........................................
It looks like I will have to pay them to be considered. It has a cooldown of seven days. Check back in a week to see if my bribe works.
I worked on this specific task myself, using Claude, and I was not blocked (although I did not succeed!)
Can you be sure that the model was not constrained for you, regardless? Since they can’t help too much with cyber, they might avoid some routes to not get completely flagged and that could influence for not finding the solution.
I have seen behaviour a little like what you describe, in other tasks. When I was cracking the keys for Sony's camera firmware updates, Claude steered away from an obfuscated library, seemingly because it thought it might be related to licensing checks. It was indeed related to licensing checks, but also firmware update decryption. I had to explicitly tell Claude to write a deobfuscator for it, which it did, and eventually I had the keys.
I later got Claude to crack their licensing checks too, by convincing it to write a "license checker" program, and then a second "test case generator" script for the license checker... (Opus 4.8 was smart enough to see what I was doing, but Opus 4.6 complied)
A form of being blocked is sandbagging the models
I saw no evidence of sandbagging. And since it was an open problem, there'd be no reasonable way to "steer away" from the correct solution.
Same. I tried it just a few days ago. This news made me almost jump out of bed.
Same!
I feel like history is repeating itself - to many people AOL, Compuserve, MSN, et al seemed superior to "the web" in the early 1990s because they dumped money into making it slicker, licensed content, provided guardrails, made it easy for non-tech people... They had slick ads, were cultural phenoms, and people would take sides with the companies and think their arguments were strong and even obvious!
I'm skeptical that this particular task would get blocked, though I could be wrong. But I agree that the moat is a problem in the general case.
I don't know about cryptography, but if you try anything in organic chemistry or biology, you virtually always get flagged and downgraded.
As this is going on, Anthropic has opened up their own biology lab.
> https://www.reuters.com/world/anthropic-quietly-sets-up-biol...
There is very much a situation where people in frontier labs have access to exceptional capabilities that are being kept from the rest of us, and they're using it to strip-mine technical and scientific problems.
I ran into this situation and was able to work around it by running a jailbroken Chinese model that turned out to be quite capable. I think it was Qwen.
Strip mining is a great phrase here, thank you
4 replies →
> There is very much a situation where people in frontier labs have access to exceptional capabilities that are being kept from the rest of us, and they're using it to strip-mine technical and scientific problems.
If these capabilities were available to the public, someone somewhere would make a biological weapon or create a virus we've never seen before that could kill many millions of people.
Stanford researchers have demonstrated it’s possible [1].
[1]: "AI creates 16 new viruses from scratch, showing promise for drug resistance and drawing warnings about potential for misuse" - https://www.cnn.com/2026/08/06/health/ai-viruses-bacteriopha...
3 replies →
Broadly, the downgrading is still way too heavy-handed and clumsy. As of yesterday “How easy to crack is WPA2” downgraded to opus 4.8. Granted I could probably have phrased it differently and gotten a response, but the point is I shouldn’t have to. It can’t simultaneously be smart enough to reverse the ECDSA curve seed and not smart enough not to destroy the world by answering a trivial question with a scary word in it.
At minimum we don't have access to "Claude thoughts" like the ones the article highlighted, right? That's part of the hidden reasoning?
Correct.
Fable 5.1 has downgraded me in the past week alone for the crimes of describing a coordinating server as "the C&C server" ([cyber]), for doing some statistical analysis on some toy Vigenère-ciphered text ([cyber]), and for asking it to help me find patterns in some semi-random text ([bio]), so it is fair to say I am not skeptical.
I have access to Claude's "defensive" cyber model and it appears to take no issue with the prompt, but I suspect that is because it uses today's publication as the basis for an answer.
In the past my attempts to work on crypto related problems have been met with needing the higher tier of "offensive" access.
It doesn't matter what is blocked today, it matters what is blocked tomorrow.
If we allow these providers to promote regulatory capture they become the arbiter of what anyone is "allowed" to do
I think when the open source models catch up (likely soon) we'll long for the days when companies could control it like this. I could easily see a teenager with openclaw and a single prompt causing millions of dollars in damage. Multiply that by the number of stupid people in the world and we have a problem.
I know Anthropic models have a lot of insane, paranoid filters, but I don't remember them blocking cryptography work last I checked (some 3 months ago now).
I routinely get rejected on elliptic curve related requests; often I can talk Claude back into some work on it. But it is surprisingly sensitive to talking about research cryptography in my experience. I’m not in their cyber program.
3 months ago is like years ago in the current AI landscape
Three months ago is about when I gave up on trying to use their newer models for almost any task without getting blocked.
But EC never seemed to cause trouble somehow.
The cyber filter blocks really basic stuff all the time. People who are pro that bs should be forced to only eat with a spoon for the rest of their life.
For me it refused to move Claude transcripts from point A to point B on my local machine. It kept getting flagged as reasoning extraction or something. The line of "It's my machine, I can move files wherever I want" shouldn't be crossed. Someday file managers are going to go extinct and LLMs will organize everything, and they shouldn't choose what files you can and can't access on your own computer.
If you have a variable `const char *shellcode` in a fizzbuzz function, it is enough for it to lose its mind.
1 reply →
> moat widening between the model providers, their government partners, and the public.
Part of me wonders whether the NSA/other TLA has either (a) an unrestricted model from a lab with less scruples like SpaceX, or (b) an in-house model, either abliterated or with data they've bought themselves.
Anyone can do this kind of work with a model as simple as GLM 5.3 or Kimi K3
How do you know this to be true? I am not disagreeing; I genially need to understand why or if others are seeing blockers that I don't see or if I am seeing blockers that they do not see.
Are their guardrail benchmarks or anything of the sort that we can use to see where the bifurcation of permission/capability occurs?
Why would you be blocked? I cannot imagine why you would be for this specific task unless the supposed backdoor turned out to be real.
As much as I dislike the guardrails, they are absolutely irrelevant to this task, and you will not get blocked.
Moat must have been written more times in the last 12 months than in all of history before that time.
Open models are just as quick to refuse to do stuff, in my experience.
Compute has always been gated. This is nothing new. If we look at the HPC space, many of the largest clusters were restricted access and used by governments organizations. Even the ones with "public" access, not just anyone could get access to it.
JFYI, this is not true. I do stuff like this all the time.
Whenever I say that the rich have access to much more capable and unrestricted models I get downvotes. Yet here we have another example.