Comment by jiggawatts
3 days ago
> while you could ask your LLM to do that you aren't going to run your bank on the result.
Why not?
I feel like we're entering a new era of prejudice against not a category of humans, but against non-human intelligences.
The design patterns for distributed and fault-tolerant systems are well-known and established in the industry. Both humans and AIs are familiar with them!
So if you sketch a design for the AI to follow, establish the rules in AGENTS.md, have a robust test suite, use a frontier model dialed up to eleven, etc... why not rely on the LLM output?
At the end of the day, humans are not without fault either.
I've been wading through some legacy "pre-AI" code recently and it has more bugs than a rainforest! Static fields used incorrectly, causing data races. Floating point types used for money amounts. JavaScript and SQL injection up the wazoo. Wildly unsafe password handling. So on, and so forth. This is the norm for most human-written software, not the exception.
As a proof-of-concept, I tried an AI rewrite of one such legacy app[1], and it is not bug free, but it notably has fewer bugs than the original. Different bugs, sure, and I'll have to iron them out after a round or two of UAT, but I'm honestly more confident with what I got from the chatbot than the code inherited from humans.
[1] Deals with money, but admittedly at a much lower level of risk and consequence than a banking app running on a mainframe.
Because you know that the current one works. If you have a bank running on COBOL (or whatever), you've had that for 30+ years now, so while it might have bugs, you know what they are. You don't know what the LLM output is. Hence back to my original point: writing the code is not the hard bit. Making yourself (and your CEO etc) comfortable to put that into production is one of the hard bits.
> Because you know that the current one works.
What do you even mean by "works", specifically?
> it might have bugs, you know what they are.
Okay, so it doesn't work, you know it doesn't work, it's just that you accept the specific ways in which it doesn't work.
I've lost track of all the myriad stupid ways in which these ancient systems are hugely ineffectual without even being outright faulty.
Like airline tickets where your name is printed as "LASTFIRSTMR" in all caps and no spaces because their systems are ancient beyond belief.
Similarly, my bank statements are security-critical, because anyone with a copy of my credit card details can pull money out of my account without my express authorization. But...
... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
If you look at it from the perspective of someone used to modern web security, then you realise that banking is a raging tyre in comparison. Banks literally just accept a certain rate of criminal activity and "price that in", reversing transactions when asked -- which itself can also be a criminal activity. They just shrug their shoulders.
"What can we do about this?" -- says the people that have tried nothing and are all out of ideas.
Rewrite it. The whole thing.
Use an actual database, something made in the last three decades instead of half a century ago.
Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Etc.
I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.
> What do you even mean by "works", specifically?
It runs and accepts people's payments, which means you're not on the front page of the newspaper (not in a good way).
> ... because they're stored in terrible ancient mainframe databases, the text fields all have tiny maximum lengths. Hence they're all abbreviations. Attacker-controlled abbreviations without any authenticity assurance of any kind!
They also have to go through payment networks which are very often the limits on those things. So yeah, it sucks, but just fixing one DB isn't enough - the whole thing has to get upgraded.
> I have no idea who actually transfers money out of my accounts! There are no URLs, no metadata, nothing to actually confirm the identity of the other party. Every field in a transaction record is 100% attacker-controlled and unverified by my bank.
There is a little bit, but not much. Again, if these transfers are happening via card, it's all a terrible old fixed-length setup. Would be great if it was better, but you need Visa and Mastercard to upgrade as well. And of course there _is_ verification - most banks don't do a great job of surfacing this but they know if they've verified a PIN or CVC, or if it was contactless (in which case it _is_ unverified, but society realised we prefer the convenience there).
> Use cryptography. No, not crypto coins! I just mean a bog-standard algorithms like public-private key signing so that it is possible to confirm the source of transactions.
Obvious question then: You've made a card transaction. It is signed with the other party's private key. What does that buy you? How do you attach trust to this key? Whose is it - the payment gateway or the merchant?
> I would much rather have something generated with the assistance of a modern LLM than what we have now, which is security holes big enough to drive a panamax container ship through.
Sure, and if the LLM can rewrite enough of this system to get what you want, there's heaps of room for improvement. But this is orders of magnitude bigger in scope than rewriting your one old bit of COBOL software, it's systematic.
1 reply →
How did you conduct this rewrite? Did you hand the AI some specs, some tests, the existing code?
I feel like AI has dramatically changed how complete rewrites can be considered, especially for long-lived, legacy projects.