Comment by chromacity

> Do we, really?

Yes, or pretty close to it. What we don't know how to do (AFAIK) is do it at a cost that would be acceptable for most software. So yes, it mostly gets done for (components of) planes, spacecraft, medical devices, etc.

Totally agreed that most software is a morass of bugs. But giving examples of buggy software doesn't provide any information about whether we know how to make non-buggy software. It only provides information about whether we know how to make buggy software—spoiler alert: we do :)

> No matter where I look, up and down the stack, across different OSes and tech stacks, there are bugs.

I’m not sure I’d go quite as far as GP, but they did caveat that we often choose not to write software with few bugs. And empirically, that’s pretty true.

The software I’ve written for myself or where I’ve taken the time to do things better or rewrite parts I wasn’t happy with have had remarkably few bugs. I have critical software still running—unmodified—at former employers which hasn’t been touched in nearly a decade. Perhaps not totally bug-free, but close enough that they haven’t been noticed or mattered enough to bother pushing a fix and cutting a release.

Personally I think it’s clear we have the tools and capabilities to write software with one or two orders of magnitude fewer bugs than we choose to. If anything, my hope for AI-coded software development is that it drops the marginal cost difference between writing crap and writing good software, rebalancing the economic calculus in favor of quality for once.

> Do we, really?

Formal verification to EAL7[0] in theory, as long as your requirements are correct.

In practice I'm not aware of any bugs being discovered in any EAL7 software, but it's so expensive there isn't a lot of it.

[0]https://en.wikipedia.org/wiki/Evaluation_Assurance_Level

> Do we, really?

Yes. There’s a ton of lessons learned, best practices, etc. We’ve known for decades.

It’s just expensive and difficult. Since end-users seem to have no issue, paying for crud, why bother?

>>> often due to factors outside of their control.

That’s the beauty of OSS - the level we could write code is way less than the level the culture / timescale / management allows. I recently saw OSS as akin to (good) journalism for enterprise - asking why is this hidden part of society not doing the minimum (jails, corruption etc).

Free software does sooo much better compared to much in-house it is like sunlight

> > We know how to write software with very few bugs

> Do we, really? Because a week doesn’t go by when I don’t run into bugs of some sort.

I mean, we do know how to do it, but we don't because business needs tend to throw quality under the bus in exchange for almost everything else: (especially) speed to develop, but also developer comfort, feature cram, visual refreshes, and so on always trump bugs, so every project ends up with bugs.

I have a few hobby projects which I would stick my neck out and say have no bugs. I know, I'm going to get roasted for this claim, but the projects are ultra simple enough in scope, and I'm under no pressure to ever release them publicly, so I was able to prioritize getting them right. No actual businesses are going to be doing this level of polish and care, and they all need to cut corners and actually ship, so they have bugs. And no ultra-complex project (even if it's done with love and care) is capable of this either, purely due to its size and number of moving parts.

So, it's not like we don't know how to do it, but that we choose not to for practical reasons.

We do.

The issue is almost always feature management.

Back in the days I was making Flash games, usually a 3-5 weeks job, with no real QA, and the project was live for 3-5 months. Every time I was ahead of schedule someone came with a brilliant idea to test few odd things and add couple new features that was not discussed prior. Sometimes literally hours before the launch.

Every time I was making the argument that adding one new feature will create two bugs. And almost always I was right about it.

Fast forward and I'm working for BigCo. Few gigs back I was working for a major bank which employed supper efficient and accountable workflow - every release has to be comprised of business specific commits, and commits that are not backed by explicit tickets are not permitted.

This resulted in team having to literally cheat and lie to smuggle refactors and optimizations.

Add to that that most enterprise projects start not because the requirements were gathered but because the budget was secured and you have a recipe for disaster.

I started to write a comment about how low they probably were able to bribe people for but found this article [0] which put the number higher than I expected:

> One of the core LAPSUS$ members who used the nicknames “Oklaqq” and “WhiteDoxbin” posted recruitment messages to Reddit last year, offering employees at AT&T, T-Mobile and Verizon up to $20,000 a week to perform “inside jobs.”

That said, this is but one instance and I'd imagine that on the whole they are able to bribe people at much lower numbers. See also: how little it takes to bribe some government officials.

[0] https://krebsonsecurity.com/2022/03/a-closer-look-at-the-lap...

And because it is surprisingly difficult to distinguish between 'oops' and 'malice' a lot of the actual perps get away with it too, as long as they limit their involvement. In-house threats are an under appreciated - and somewhat uncomfortable - topic for many companies, they don't have the funds to do things by the book but they do have outsized responsibilities and pray that they can trust their employees.

> they can afford to simply buy your software dependencies, or to offer one of your employees some retirement money in exchange for making a "mistake".

Orthogonal, but in similar spirits: the FAANG part of big tech paying less, doing massive layoffs, and putting enormous pressure on their remaining engineers might have this effect too in a less directly malicious way.

Big tech does layoffs, asks engineers to do "more". This creates a lot of mess, tech debt, difficult to maintain or SRE services. Difficult to migrate and undo, difficult to be nimble.

These same engineers can then leave for startups or more nimble pastures and eat the cake of the large enterprise struggling to KTLO or steer the ship of the given product area.

Keep in mind, the billionaires seem to think they can crash all this into the ground, and some how survive by buying their own miltaries.

The scale of how society works is lost on the greedy

Should something like a WordPress server not have a domain allowlist for outbound connections? Does WordPress need to connect to arbitrary domains?

That is a never-ending game of whack-a-mole. There are infinite places to put command and control data.

>if they're rational (an assumption that is debatable, of course)

Feels like crime is an almost perfect simulation of the free market: almost/ all of the non-rational actors will be crowded out by evolutionary pressure to be better at finding the highest expected values, where EV would be something like [difficulty to break in] x [best-guess value of access].

To be fair, the recent Axios supply chain attack was North Korea based, and probably cost them very little money. So it illustrates that you don’t have to “spend a lot of money” to get into our systems.

Especially because you can potentially use a model like Mythos to figure out how to hide (from humans, at least) a deliberately created vulnerability.

Rogue nation = not under strict USA control.

If we wanted to treat words literally, the true rogue nation is USA. The only nation on earth to have actually dropped nukes on people. Have been prooved to spy on the entire world population. Plants coups around the globe. Invades any country they fancy in the name of democratization.

If that ain't a rogue nation I don't know what is

First, I'm not dismissing anything. I'm just saying it's not the most significant concern. Second, Mythos doesn't create "new ways". You already have plenty of vulns to go after, and you can write exploits for them (or pay someone). It just lowers the cost / commoditizes the toolkit. It's not the first time it has happened - the trend goes all the way back to Metasploit or before.

And again, I'm not saying it doesn't matter. All I said is that it's probably not the #1 thing to lose sleep over.

>we at least have the technological foundation to escape oppressive, corrupt and dysfunctional governments financially

Who is we? How many transactions of any cryptocurrency was either done to buy bread and butter?

IMO the thing that AI will change is the type of target. It's reasonable to assume that if you launch a website for a small business nowadays - sure, you'll get phishing attempts, port scans, attempts to submit SQL injections into your signup forms, etc.

But you won't get the equivalent of a sophisticated actor's spear-phishing efforts, highly customized supply chain attacks on likely vendor data, the individualized attention to not just blindly propagate when a developer downloads a hacked NPM package or otherwise gets a local virus... but to log into the company's SaaS systems overnight, pivot to senior colleagues, do crazy things like update PRs to simultaneously fix bugs while subtly adding injection surface areas, log into configuration systems whose changes aren't tracked in Git, identify how one might sign up as a vendor and trigger automatic payments to themselves with a Slack DM as cross-channel confirmation, etc.

The only thing holding this back from hitting every company is risk vs. reward. And when the likelihood of success, multiplied by the payout, exceeds the token cost - which might not happen with Mythos, but might happen with open source coding models distilled from it, running on crypto mining servers during times that minting is unprofitable, or by state actors for whom mere chaos is the goal - that threshold is rapidly approaching.

It reminds me a bit of the Segway hype "they'll build complete cities around these".

Sure, you can find problems faster, but it's not like they'll find 20 NEW classes of bugs.

Yes, on the one hand, they enable a lot of shady illegal business, but in the other hand, they also destroy the environment while doing it, so it's really a toss up whether cryptocurrency is good or bad overall!

Crypto has been an awful development in many ways, but I happily welcome it when it has made malware so much more benign to me. The last malware that affected me personally was a crypto miner worm, and the one before that was a crypto wallet stealer, neither of which affects me at all as I don't meddle with crypto.

I don't know the statistics, but it seems like it's way more profitable for the grifters to target other grifters instead of taking over my machines and extorting me. Or maybe I just got lucky.