Comment by wglass
6 days ago
It's crazy enough this will be fixed soon.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
> After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
Banks and financial institutions are the same. If they haven’t issued you a monthly/quarterly statement yet, they can just apologize and tell you the numbers are wrong please wait for the statement. But it is a major issue if an actual statement has the wrong numbers.
Reminds me of working for a cable company and being told that even if we screwed up and stole from the customer the look back period was only a few months and if we found an error from before that we weren't supposed to correct it.
There's a certain obligation on both sides of a contract to pay attention.
If you're not watching your billing, and then try to claim overcharging a year later, you'll get a lot less satisfaction even from regulators or judges than if you notice it when (or soon after) it happens.
2 replies →
A couple of my coworkers think I’m nuts for watching cost explorer so closely but
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
I've personally noticed and saved multiple $xx,xxx monthly cost billing spikes just by take a daily glance at our cost explorer. I'm in the AWS accounts every day doing investigative work anyway that an extra 30-60 seconds is trivial.
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
> 2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Unfortunately not a choice at my organization
> yacht money
Huh? Apparently Amazon has never paid dividends, has done much less than 1% buybacks, even as CEO Bezos was getting less than an engineer in total comp (but did get loans or something).
Apparently Bezos owns about 9% of common stock - no dual class shite.
So your yacht spending sounds like a hallucination: if I'm missing something, please correct me.
Amazon's weight: about 3.7–3.8% of the S&P 500. So he isn't even that overweighted (especially as the market doesn't like owners selling down - bad optics).
Bezos has spent about 28 billion for Blue Origin by selling Amazon stock - not using shell games. He has comparitively spent a few percent of that amount on superyachts.
Do you just watch for trends? Curious to know your process to verify if the cost is right or wrong.
>> It's crazy enough this will be fixed soon.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....