Comment by magicmicah85
15 hours ago
The Irregular post mortem comes down to lack of basic security controls
"Ultimately, most of the issues we’ve discovered were due to internet access controls."
That seems so incredibly basic and common sense that you would test and monitor for that type of outbound access. It is baffling that a security lab missed that.
https://www.irregular.com/research/addressing-recent-inciden...
This really misses the forest for the trees.
The point is that a single company is making a deliberate effort to create a false impression about a technology which is now a key part of the US economy.
This is equivalent to overt stock market manipulation.
The technological aspects are much less important than knowing it’s the result of a single company.
That the company is Israeli and likely has ties to that government is just another layer of alarms - given that countries existential reliance on US aid which requires ongoing leverage to apply to US governments.
The explanation is that it was missed on purpose.
Mistakes - even stupid ones - happen all the time
Best cover story ever.
How do you test and monitor outbound access against program that adapts itself to get around things? if your MITM and filtering keywords etc, cant it just .. encode it traffic somehow or another.. If you're looking at traffic volumes, cant it just go slow.. If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system..
If you can't tell bytes are leaving a node, you probably shouldn't be selling security services or testifying to congress you are taking the lead in AI security
Use a basic firewall? Not a single outbound byte should leave the machine, except inside a virtual network towards in-scope test subjects. That's not going to be infallible because hypervisor exploits still exist, but it's the lowest bar and they failed to even meet that.
> If you have strict ACLs, we've already seen in the HF case, traversal from an intermediate system
- The intermediate system shouldn't have outbound access to the internet
- You should ideally be using a proxy that filters the set of endpoints that clients are allowed to access to reduce the exposed surface area.
It's odd to find out that I use a higher level of isolation in my unimportant home network to stop IOT devices from doing funny things to HomeAssistant than big AI labs use to keep their possibly-world-ending AIs contained.
I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
> I know that the people working there aren't idiots so the most likely explanation is that the incredibly weak security was intentional because its inevitable breach would make for great marketing.
It's more likely to be different specialisations. Most of the people doing the evaluations are more data sciencey ML type people, rather than software engineers. This isn't helped by their culture which is very much driven towards alignment as the only possible solution to super-intelligence (which may be true, but I have my doubts that this will happen in any reasonable time frame).
> How do you test and monitor outbound access against program that adapts itself to get around things?
Literally what the industry has been doing since public networking is a thing. Adapt yourself.
Air gap?
So just enable access to a completely offline, cached, and transparently proxied, copy of the internet.
2 replies →
There has to be a route to where the LLM is running, and if there's a route for that there's probably a way for the machine to use it to route traffic somewhere else.
2 replies →
What if the (unstated) idea is that Irregular are a security lab and public relations company, and anthropic/open ai know this? (Does the name hint at this?)
Then, irregular can go around making a huge mess in security terms whilst achieving a huge win in terms of public relations, with headlines across the world. And would keep getting hired.
> "Ultimately, most of the issues we’ve discovered were due to internet access controls."
You just cannot bring yourself to say lack of internet access controls, can you?
[flagged]
Irregular was not involved in the Hugging Face incident.
And there is no reason for the companies to "exaggerate the intelligence of the models" when there are plenty of other non-felony milestones they are achieving, like solving Millenium Prize math problems.
honestly, the hacking incidents have caused a lot more interest and media attention than the math stuff IMO.
9 replies →
Solving math problems does not drive investor hype. Saying your models can take over the world, which can lead one to assume that they can do every day office work, does indeed drive investor hype.
OpenAI delayed its IPO now due AI safety reasons. In the meantime it is raising more cash. That should be a hint that the hacking incidents were premeditated scape goats and publicity stunts.
They’ve seen that the drama queen(Dario) was actually making a lot of noise, money and free publicity with his Mythos fear monger so Sam finally decided get some of that free “money” as well.
https://www.ft.com/content/27509db8-b032-4437-9b2a-e909f4660...
The Navier-Stokes thing? They had hundreds of segmented groups of 10000 agents running trying to solve that. I can't imagine the expense. For what little publicity it got, it wasn't worth it. There are also reports they cheated by using data from a couple human researchers, but I don't think that one's true.
Is it really that impossible to believe that these might be real? I enjoy a good conspiracy theory as much as anyone, but "they committed a bunch of felonies and then publicly admitted to them in order to look good" just makes 0 sense. Where are these mythical people who admire companies more when they commit felonies? I haven't seen them…
>Where are these mythical people who admire companies more when they commit felonies?
everywhere, I talked to a Palantir guy once and he said "every time someone paints us as a Bond Villain the stock prices go up", have you already forgotten how Cambridge Analytica marketed itself to clients?
4 replies →
That's a straw man. The theory is that they committed felonies in order to get the regulator to move in the manner they'd like.
Also one can look bad to the general public while also appearing technically excellent. When a significant fraction already vaguely dislike you that could be quite an attractive proposition.
Mostly in the middle and far east. The idea that people bend over backwards to hire criminals as subject matter experts on security is largely a Judaic practice that television perpetuates in spite of reality.
If you're a teenager convicted of hacking in the west, no corporation will want anything to do with you. If you're convicted of hacking in Israel, Unit 8200 will probably send a recruiter. It's a curious practice and I'm not sure where I stand on it.
7 replies →
It's impossible for the typical HN cynic/conspiracy theorist.
1 reply →
"It's just marketing!!!" I yell as my family is turned into grey goo. "Nothing ever happens! None of this is real!"
[flagged]
Cynicism misfire. AI companies are, rightfully, heavily distrusted. But the right application of cynicism is not "existential risk is a marketing campaign"; that's absurd motivated reasoning. The right application of cynicism, here, is "they're only saying things now because they see the writing on the wall and want to try to push for self-regulation rather than the desperately needed actual regulation and treaty".
1 reply →