Comment by akdev1l
6 days ago
Not even tests but just some basic anomaly detection lol.
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
6 days ago
Not even tests but just some basic anomaly detection lol.
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.
In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.
COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.
Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.
Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.
1 reply →
Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.
If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.
That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.
2 replies →
> > everything is driven with sticks instead of carrots
> this is not the case
> [describes all sticks and no carrots]
> COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening.
Absolutely not my experience at AWS.
All the teams I was on treated them as "not a big deal", kind of a non-punitive exercise in technical writing, and the COE was always assigned to be written by an engineer who was not involved in causing the COE.
Also, the kinds of issues that did or didn't lead to COEs appeared to be largely random. I was considered to be an extremely good operational trouble-shooter on the team where I spent most of my time at AWS, and I was never able to predict what an L7-8 manager would decide was COE-worthy.
3 replies →
> Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again
And this surely works out great and completely prevents any issue, and this "inaccurate estimated billing data" issue this submission is about is both the first and last time this issue ever happens at Amazon?
[dead]
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
Yeah but this also negates the argument that people gotta use it or they get left behind.
If it takes no skill or intelligence then nobody can get left behind because it's very fast to get back in.
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
I am former AWS and this is pretty accurate.
The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).
3 replies →
Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
3 replies →
Yes, I am a former AWS employee.
I got put on Focus because my "contributions were not coming through" to leadership.
"Former" seems to an important detail here.
3 replies →
> take the 3am on-call and be the hero instead
Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"
I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.
The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
1 reply →
How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?
1 reply →
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.
I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.
I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)
It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.
> there's definitely this sort of monitoring in place. Surely somebody was paged.
Well you’re half right. Either there wasn’t monitoring for this or if there was monitoring in place that’s not what caught this, because the page originated from a customer support ticket.
Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
They've already got anomaly detection: their users.
I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
If only there was some way to get anomaly detection services [1] inside of AWS...
1: https://aws.amazon.com/what-is/anomaly-detection/