Comment by donavanm
6 days ago
Ive dealt with this error at AWS. It’s a unit error. In my case we _meant_ to charge like 5¢/GB, but missed the unit (GB), and then the billing system defaults to bytes. 5¢ per Byte of data transferred meant some customers were seeing MM bills within hours. Got paged by support around 2am, had it fixed and amendments issues by 3-4am, apology emails shortly after.
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
> Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
If you support multicurrency billing, then have the tests bill a test customer in XTS (ISO 4217 code for test currency).
6 replies →
But these aren't the right services where the test should be, right?
There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".
It's the third service that multiplies these things that failed. Where are the tests for that?
6 replies →
> well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime
You don't have a non-production environment where you can have billing that doesn't involve real money?
the cynical take is of course that the tests were written by Claude and not validated.
Unfortunately "automatic generation" of unit tests have made a big part of the value of unit tests ("I am telling you what I expect here") disappear.
2 replies →
[flagged]
Not even tests but just some basic anomaly detection lol.
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
34 replies →
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.
I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
1 reply →
Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
They've already got anomaly detection: their users.
I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
If only there was some way to get anomaly detection services [1] inside of AWS...
1: https://aws.amazon.com/what-is/anomaly-detection/
>no tests?
Earlier this week my slopservant implemented several comprehensive changes to a codebase.
It also wrote extensive tests to verify the correctness of the changes.
A few days later I was working on something else and realized, everything had been implemented backwards, in a way that was nonsensical and also completely pointless.
The many tests it had written were just confirming the LLM's idea of correctness, which turned out to be... completely incorrect.
I laughed when I realized, if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
Not sure what lesson to take from that (except read the damn diffs, obviously — it was a hobby project okay ;), but it seems like the more reliable this stuff gets, the more we expect it to work properly, the more risky it becomes.
https://en.wikipedia.org/wiki/Normalization_of_deviance
I'm struggling with this on a hobby project now. I'm torn between continuing the fast pace of development and taking a pause and checking the entire codebase.
The more features I add now will likely make the inevitable refactor much harder, but adding new features is so easy that I want want extend this illusion of productivity just a little bit longer!
1 reply →
> if I had been using Rust, or indeed, formal verification, that wouldn't have helped at all — it would have just written a mathematical proof, proving the correctness of the wrong thing!
This is a well-known issue in formal methods, often referred to as the "formalization gap". It's pretty much an intractable problem, i.e. all we can do is manage it, not solve it.
In fact this issue exists in many fields, often under the name "specification gap" or "specification problem". You'll find versions of that discussed inv economics, architecture (of buildings, not software), etc. In architecture and construction, a high percentage of mistakes that result in lawsuits turn out to be due not to construction, but to specifications that were incorrect in some way.
The bottom line is that there's no perfect formal/mathematical way to go from intent to proof.
The solution is a skill that explains to the LLM, with a fresh context, to NOT read source files, and to only use XYZ tooling to extract documentation and function signatures, etc. without implementation details.
Then it can write tests based on the interface, and that is usually a lot more solid.
2 replies →
The lesson is: write your own code, it's not that hard.
write the tests yourself. Or at least the sketch of the test (to get filled in later). Make a commit of what you do so you can then look at the changes.
If you write the tests the agents have an easier time writing the code!
This reminds me of a discussion a few months back from a BSD maintainer who had done a lot of volunteer work for AWS over the years. I think he might have even been the person who alerted them to the insecurity of IMDSv1. There was a sense that AWS might have had great talented developers at the time, but they clearly didn't really understand the domain of running a hypervisor service exposed to the public. This feels like a similar situation, where I'm sure they "test" what they know to test, in the way they know, but compared to a bank, well, it wouldn't surprise me the least if no one at AWS ever even thought to ask banks how they handle things like this. Instead of testing that a process works the way you specified it, you need to make sure you even have the right specification in the first place by consulting with prior art and ensuring whatever you translate into software reflects the legal reality of the process you're trying to encode.
There's a similar thing with physical processes. My wife works in geointelligence ground processing and encountered this when her system was expanded to support SAR collections instead of just visual spectrum. Software that passed all of its internal tests was passing nonsense collection parameters because the developers didn't understand the difference between energy collected to a sensor cell reflected from the sun versus energy reflected from your own active scan. The capabilities and limitations aren't the same. You can perform the same processing. The bits on disk won't care. But if the process you encode doesn't accurately correspond to the physics of reality, you're producing nonsense. Memory-safe, syntactically-valid nonsense, but nonsense nonetheless.
This seems to happen quite a bit with software companies.
I have bad news if you think banks are the pinnacle of cyber security practices...
Maybe some rare few are, but by and large they generally stink.
They tend to give that impression because they hire tons of auditors and panjandrum, to hassle their suppliers, but internally they are winging it like the rest of us.
2 replies →
we invoice clients monthly. if we ever sent one a bill 2^30 times too high, I don't think "estimated charges, no action required" would save the relationship :D
Who needs tests when you have vibes?
It's just like in Superman III
It’s fractions of a penny Peter.
I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
The problem with an anomalous AWS bill is that I don't know if it's because of a bug on their end or because of a goof on my end, eg did I accidentally leave on some giant compute for a month or something. With a billion in my bank account I at least know that I definitely didn't put it in there.
If I experience a bug when transferring money (switching a digit or some such) they’ll shrug their shoulders and say there is nothing they can do. No amount of proof will push them to increment and decrement some integers over there.
They transfer money to me. Their mistake: there is nothing I can do. Rules for me but not for thee?
I’d be concerned as it throws into doubt their entire accounting mechanism
What could a banana possibly cost Michael, ten dollars?
...in the off chance that one in 100,000 will pay their bill instead of dropping dead from a heart attack (perhaps because there's a heartless OpenClaw running their accounting department), it's all worth it.
Why would you think there are "no tests"?
We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
10 replies →
I wonder if AWS billing still uses CSV files for passing data around.
IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.
Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.
Sure, and I would expect the COE called out the operational aspects. No one should have been in a position to be able to trigger that bug trivially (I don't remember if it came from a service, or if they injected the metering record themselves somehow, or quite what. Way too long ago for that)
Most engineers with more than a few years of experience know better than to directly try a stunt like that in production, but they do often get there through painful personal experience. I just wish I hadn't been one of the many that got to suffer from that engineer choosing to learn at that particular moment!
Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.
Eh… If I can bring prod down simply by sending CSV with a comma in a field to a prod endpoint, whoever has come up with the most naive CSV parser possible running in prod is responsible for the outage.
1 reply →
Give that engineer a raise and an official title of "chaos monkey".
On the scale of AWS, you want your production system to be poked at and pushed against. Obviously not something to implement out of the blue in your production if you've never had it. But certainly worth considering.
Even on a smaller scale, it makes a lot of sense to have some deamon randomly bringing nodes down, shut off a database, fill a disk up, etc etc. In production.
Because that brings awareness to all engineers that the stuff that "can happen", will happen. It makes for a culture of defensive, considerate engineering. A culture where someone who finds bugs or holes in production is rewarded, not punished.
A culture where having some automated management for nodes is not a ticket "at the bottom the backlog", but a crucial attribute when picking or engineering the hosting system. That having redundancy in critical services like a database, is not an expense, but actually a cost saver, because chance of critical services failing is 100%, rather then some made up gut-feeling-chance. That the error handling and tests for when a module cannot write a file to disk isn't something that "this senior who left 8 months ago overengineered", but a pattern to be consistently implemented¹. etc.
---
¹ It is one the things I love about rust, that it has built in enforcement for you to deal with all the esoteric errors (e.g. https://doc.rust-lang.org/std/io/enum.ErrorKind.html#variant...) that can occur. Sure, you can write `File::create("foo.txt").unwrap()` but even then, you have made a conscious decision, clearly "documented" that you will crash the program - i.e. postponed the decisions on what (groups of) errors must be handled in what way, to a time you have the information to make that decision. Same design decision with go, where errors are explicitly to be handled by the programmer.
I got known as a "Chaos Monkey" at AWS, and have sort of carried on that title. It's not exactly accurate, but tech stuff just breaks around me, and it's very, very rarely my fault. I'm pretty cautious and resist any urge to "I wonder what happens..." with anything that could possibly have an effect beyond me.
The "Chaos Monkey" effect at AWS was so pronounced you could literally see on our sev2 count graph when it was my on-call week, because I'd get dramatically more sev2s than any other engineer.
The service I worked for at AWS was amazingly stable and reliable. A large majority of the alarms that ever fired, fired because of problems with another service we depended on in some way. This billing thing is a good example. I was on-call when there was a major SQS outage in a region, through a DynamoDB outage, S3 outages, all sorts of stuff. We used to joke that it would probably be a net benefit for AWS if I wasn't on-call, just so my spooky-at-a-distance wouldn't happen.
I eventually lost my "most sev2s in a week" record toward the end of my time there when someone else was on-call and DynamoDB had a major outage. DynamoDB held all of our metadata so everything broke and every alarm we had fired over the space of a few hours.
CSV files are widely known to be used by the most frugal companies so of course it is.
I've worked on a complex bookkeeping SAAS. Our "CVS" im- and export was the most used feature by far.
Yet it was implemented on a thursday afternoon by a junior, using the first "library" that popped up in a google search. And over the years, leaked into every corner of the application. Unaffordable technical debt.
I cemented this experience into memory and now, in every new gig where I have to do "CSV", I isolate it, abstract it, overengineer it. Hell, last year I even built and launched a dedicated e2e tested csv-export service instead of just `from csv import writer` and call it a day.
Because CSV is the "interface" that will bring your system down if not properly designed.
Have you seen what RDS costs‽ AWS couldn't possibly afford that!
It was the same old kona files for metering until at least a ‘23-24 when I last did metering/pricing work. “Luckily” the size of the EC2 pricing plan was so stupidly large that it was forcing improvements to how those were serialized, and how private pricing plans were created/assigned to different payer ids.
It's one of those "I hate it, but I get it" situations. Migrating metering over to a whole new format on AWS's scale is a scarily large undertaking.
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular).
Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.
But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?
Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.
Yeah someone thinking they screwed up to the tune of tens of mlllions of dollars (or billions, someone said 2^30 error); this is different to an auto accident or something. Most people can't even fathom millions or billions of dollars.
I'm stressed thinking about it, like from a theoretical standpoint.
actually $17B isn't so bad because it's so obviously an error and not something you can fatfinger. if you expected it to be $17 and it was $17,000 you'd be much more worried as that looks like a plausible error on your part
Yeah that's what I was thinking. And if someone sent me a bill for 17 billion I would just tell them I don't have it and never will, sorry.
its so far out as to be obviously not real
a smaller error by say, just one or two orders of magnitude are much more believable as a reader
Not everyone got a 10 figure bill estimates, some posts were mentioning 8 and 9 figure estimates
I've seen people have panic attacks over much smaller amounts.
Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.
[dead]
> Services emit metering values that arent directly tied to prices.
Yep. The metering ("Kona") is separated from the billing to such degree that it's basically impossible to find anyone at the company who understands both.
I remember having to work on some metering code, and trying to figure out whether it would result in correct billing that matched our service's documentation. I was basically told off by a more senior engineer for wasting my time on something that was completely tangential and outside of the "engineering" domain entirely.
This is why I always fail loud rather than pick a stupid default.
The last thing a billing system should have is defaults, this is such an insane thing.
Hey man, vibe coders gonna vine code. No wait. Loop enginering.
We don't need to do stupid things like review or plan or strategize. Put everything into LLM and let the black box take over everything! Yay! LLMs!!!!
Were there cases of finalized invoices and fund transfers while the error was in place?
The sudden loss of working capital while awaiting a refund would be extremely disruptive for even large businesses.
Are you able to just re-run the data for the timeframe, or is "fixing it" a more manual process? And what observability are you working with into the process?
What apology letters? Never got anything, not even a reply to my support tickets.
Had it been half a million dollars or something or say like a few hundred dollars?
Was this the day that gp3 EBS volumes came out by any chance?
Nope, different service. We were introducing new methods using POST and (iirc) some differential pricing based on the actual request params. Which made it a different code path, metering value, price, etc than the existing GET requests.
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.
Worst case I've found was off by 15 orders of magnitude.
One of my personal rules is to ALWAYS suffix the variables and fields with unit names:
`timeout_ms` or `rate_kbps`, NOT `timeout` or `rate`.
Unless the variable type already constrains it (e.g. Duration in Go).
2 replies →
It's not difficult to write regression tests that catch unit mistakes.
One of the Mars landers famously failed due to unit conversion errors from metric to standard.
7 replies →
Degrees, very funny.
Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.
Units and datetime will always be the bane of any professions ...
names and money movement also come to mind.
Imagine a programming language that has physical measurement unit support so this could have never happened.
Like F#...? https://learn.microsoft.com/en-us/dotnet/fsharp/language-ref...
Exactly. There are other languages that can do that or worst case you can implement it yourself (not sure how hard that is).
[dead]