Comment by reconnecting
6 days ago
By coincidence, we've been building software for five years that other plant managers are now considering using to keep track of some operational metrics. And I don't think we should put LLMs into our development cycle, simply because the data our software is built for can have critical meaning in some situations.
The insecurity in a vibe-coded web portal isn't that someone hacks it with XSS, it's that after the next vibe-coded release, some X quietly becomes −Y somewhere no one expects.
From this perspective, having no software at all might be better, or as in your case, safer.
I don't think you should keep AI away from software development for security reasons. We're in the European economic area and are completely NIS2 compliant and we use LLM's to aid in our software development for high risk systems. None of that is vibe coded though, so it's not like I disagree with you either.
As you point out this portal isn't that, but what protects us is the processes around compliance. This can't grow from X to Y because not even the CEO has the authority to overwrite our compliance gates. The EU is a tremendous help in this area since personal liability changed things completely.
> Quothling 6 months ago [1] I work in a NIS2 regulated sector and I'm not sure we can ever let any AI agent run in anything we do.
> We're in the European economic area and are completely NIS2 compliant and we use LLM's to aid in our software development for high risk systems.
Please pick one. Either way this is a nightmare level of threat to sovereignty.
1. https://news.ycombinator.com/item?id=47063153
I don't think they are opposite. We still don't let AI run anything we do. That being said, things have changed. 4 months ago I wouldn't touch the M365 Copilot thing if you paid me to, now it's basically the only AI I use professionally. Cowork changed the way we work, and I've previously spoken about how I think Microsoft sort of won the AI enterprise "war" by selling the tools and not the mode. If you'd asked me a year ago I would've been very anti AI in general. I got proven wrong. I do think it's going to be interesting to see where the EU goes with things though. Currently you can't use all features of Cowork as an example, because they are illegal in the EU. So you can't let it run your browser session for you as an example. I wonder how long that will last.
I still doubt we will ever give an AI access to run code on our systems directly though. In isolation, sure, but other than that.
This is a side note, but my personal favorite part of Cowork is that I can roll out our compliance policy to every developer as a Microsoft Teams app (no, that makes no sense to me either). So when they try to install some package that isn't pre-approved their Cowork agent won't let them and will instead explain how they might get approval. If they then continue to reference it, Cowork will even alert us.
5 replies →
> Either way this is a nightmare level of threat to sovereignty.
I agree, the EU can't become sovereign or have privacy, if the citizens constantly work against it.
Denial of Service is still an issue whether its from an external attack or spaghetti soup code bug. Outcome for your users is the same.
From my perspective it looks like were just allowing hostile developers within our environments now lol.
Your perspective is correct. We call them shadow developers, but it's a massive issue and security threat. The more tools Microsoft adds to their AI admin center (and put behind that ridicilous Agent365 DLC license) the more we see just how big of an issue it has become. People are using their personal credit cards to buy AI tools and use them wildly irresponsible.
In the big threat picture a vibe coded web tool that's not on the internet and runs in total isolation on it's own management group on the "this might get hacked" tenant in Azure is nothing though. I'd worry more about all that OT which was compromised from the factory which sits around in the energy sector. Especially because it's very easy to draw you a risk analysis that will tell you that unless you're running a Nuclear Powerplant then it makes no financial sense to secure your stuff. The audits are so rare and the consequences so low that it's cheaper to just pay the fine (if you ever get one).
Most actual security happens when someone on the ground decides that it's just too stupid that something clearly labeled "DO NOT PUT ON THE INTERNET" was put directly on the internet.
If you aren't using AI to write your code you should definitely be using it to find bugs in the code you write by hand.