Comment by vintagedave
3 days ago
What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements.
> On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary.
Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern?
More, why is this public not a court case for wrongful termination?
Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
>Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?
IME, yes.
I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything that complicated in terms of cloud infrastructure; just VMs, load balancers, some blob storage, some k8s clusters.
Stuff on Azure just breaks constantly, and when it does break it's very obvious that Azure:
1. Does not know when they're having problems (it can take weeks/months for Azure to admit they had an outage that impacted us)
2. Does not know why they had problems (RCAs we're given are basically just "something broke")
3. Does not care that they had problems
Everyone I work with who interacts with Azure at all absolutely loathes it.
But doesn’t this experience contradict what OP is saying in a way. If azure is always breaking wouldn’t that imply that changes like “adding smart pointers” are being introduced into the codebase?
I don't think it contradicts the OP. OP says the system is unreliable. Memory leaks that lead to out of memory failures for example. Smart pointers would stabilize things. (Also note that OP says their smart pointers PR was rejected).
2 replies →
As a former MSFTy it does sound weird to me too. I didn’t see what Axels level was but a lot of people work for Microsoft and not many of them can expect to email the CEO and get a response. It seems a bit like a crash out, not the first I’ve seen levied at Azure, won’t be the last. They probably think it’s a mental health episode, if you’re an important CEO crazy people will email you all the time and the staff probably filter them out before they see it. Also this is a lot of internal gossip, I would be worried that airing this publicly would impinge on future career opportunities, even healthy orgs would appreciate some discretion.
I’m sure everything he said is completely true, Azure is one of the few tech stacks I refuse to work with and the predominant reason I left.
If you’ve joined an org and nothing works the reason is usually that the org is dysfunctional and there is often very little you can do about it, and you’re probably not the first person who’s tried and failed at it.
While Microsoft is hierarchical - but it did encourage reaching out in a "flat" manner internally.
In my experience - a loooong time ago ago now - executive leadership would participate in high-level escalations/critsits for large/key customers on calls. I was just a lowly field-engineer - but over the course of nearly 4-years, was on calls about 5 times with some of the big-names from that era that everyone knows about... And they seemed to emit enough empathy with the specific customer situation to move things forward.
However - being on the "other-side-of-the-fence" (i.e. external, consulting with Microsoft customers - some of them who even spend $1.5billion/year in M365/Azure licensing) and assisting clients with issues and remediations for the last 10-years, things are no longer the same. No amount of escalation gets further than occasionally reaching some level of the product team - and it can take 8-12 months before that even occurs. Troubleshooting and deep-engineering support skills for cloud customers are typically non-existent, and the assigned resources seem to just wait until the issue resolves itself...
Never worked at a FAANG, but from what I read from their cultures I don't think a letter to the CEO from a senior engineer would go entirely unnoticed there. CEO's might receive crazy letters, but hopefully not regularly from their senior engineering staff..
Microsoft has a large PR department to put out such false impressions. The culture has changed, AFAIK you used to be able to email Bill Gates and be fairly confident he would read it, but you better be sure it was worth reading or he would fire you. Now they’re unlikely to fire you but they’re unlikely to read it either.
Senior leadership seems to be more far sequestered now, a bit like Trump, surrounded by lackeys giving them an entirely false impression of the world. That’s how they could legitimately believe they were going to bury the IPhone.
putting aside that MS is too huge to even just know about the names of your senior engineers across the globe and that the mail might have gone directly to spam
there is still the issue that this might have been classified as "a crazy letter"
a lot of the article reminds me of people which might (or might not) have competency but insists they know better and are very stubborn and very bad and compromising on solutions. The subtext of the articles is not that far afar from "everyone does everything wrong, I know better, but no one listens to me". If you frame it like that it very much sounds like a "crazy" letter.
Strictly speaking it reminds me a lot about how Pirate Software spoke about various EA related topics. (Context: Pirate Software was a streamer and confidence man who got complemented up due to family connections and "confidently knew" everything better while having little skill or contributions and didn't know when to stop having a "confidently bad" opinion. Kinda sad ending given that he did motivate people to peruse their dream in game design and engage themself for animal protection.).
Or how I did do so in the past. Appearing very confident in your know-how ironically isn't always good.
And in case it's not clear: The writing reminding me of it and having patters of someone trying to create a maximally believable writing to make MS look bad doesn't mean that he behaves like that or that the writing is intended to be seen that way.
It's more about how we have a lot of "information" which all look very believable, but in the end miss means to both: Verify many of the named "facts". And, more importantly, judge the sentiment/implicit conveyed information.
Especially if we just take the mentioned "facts" without the implicit messages and ignore the him<->management communication issues I would guess a lot of that is true.
A "Senior Software Engineer" at Microsoft is someone with a pulse and 3 years of experience (due to title inflation); so despite the "senior" in the title definitely not "senior engineering staff".
I like how caring about fiduciary responsibility is a mental health episode or personality disorder to enough people in the comments. Simply being employed gives you a vested interest in keeping an operation above board and healthy. If you have a stock plan, you have equal rights to comment on issues as some low IQ private equity chief that does an end run to manipulate a company for their own benefit. The cattle psychology of most IT workers and mid level managers never ceases to amaze me.
It is emblematic of a crash out, I’ve seen a lot of them, and I’ve thought about doing it myself. I understand the impulse first hand. I did quit over my misgivings but I did not write a blog post about it, that would have been a career ended for me. I am expected to keep secrets as part of my job.
I work for myself now, for less money, but I do get to build things to the quality level that I want.
1 reply →
In my experience Azure is full of consistency issues and race conditions. It's enough of an issue that I was talking about new OpenAI models becoming available via Bedrock on AWS and how convenient that was since I wouldn't have to deal with Azure and my colleague in enterprise architecture went on an unprompted rant about these exact issues. It's not the first time something like this has happened and I've experienced these issues first hand, so yes. I'd say reliability is a critical issue for Azure and it hasn't gotten better each time I've gone back to check.
I recall seeing some pretty damning reports from a security pentester that was able to escape from a container on Azure and found the management controller for the service was years old with known critical unpatched vulnerabilities. Always been a bit sceptical of them since then
A decent portion of Azure Web Apps internals hasn't moved past .net core 2.3
Large orgs make decisions that prioritize short-term metrics over long-term quality all the time and nobody tracks whether those tradeoffs actually paid off. The decision to ship fast and fix later sounds reasonable in a meeting setting until articles like this surface and the reality comes through clearly.
> sounds reasonable in a meeting setting until articles like this surface
No. It sounds reasonable past that. Because shipping features will make shareholders happy while an article like this will change nothing.
Microsoft Azure holds 20% of global cloud infrastructure market, 2nd largest behind AWS. Seems to be working.
I am sort of confused how NDA and such agreements employees sign would allow for an employee to post such an article without being sued by Microsoft?
Wild guess, touching this with a 10-foot pole risks validating his claims. If they sue for breach of NDA, it means his claims are factually correct, and if they sue for libel and it goes to court, they may be forced to submit documents they don't want to.
Most likely, the author was let go in mass layoff, and they forgot about NDA.
NDAs are usually signed when you join the company, not leave it.
Signing a non-disparagement agreement is often a condition for receiving severance, although I'm not sure what MSFT's policy on this is.
If they can swing it as legit whistleblowing somehow, they might be ok.
Interesting point. Time will tell.
What I meant is that it’s customary to write to the Board through the Secretary as opposed to write directly or through some other channel.
Thanks for the direct reply! I wasn’t aware it was ever customary to write to a board.
But I do see you have very clear concerns.
One thing I don’t fully follow is: how did it get from such a nicely designed system, built by Dave Cutler, to this — simply moving fast and building tech debt?
I was there when the SDET role was eliminated.
Our team of 8 SDE and 5 SDET became a team of 8 devs who also owned tons of QA frameworks. It was awful, except each tiny we deleted a test suite for not being valuable; that was awesome!
If the Azure org was flooded with senior engineers who did not have senior-level experience architecting production software... That explains a lot.
Writing to the board is not customary. When you do so, it is customary to do it through the secretary.
> What are we reading here? These are extraordinary statements. Also with apparent credibility.
I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen.
The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly.
But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way. One one team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.
It wasn't even a talent thing, Windows development skills are a career dead end outside of Microsoft. They used to be a hot commodity, and Microsoft was able to hire the best of the best from industry. Now they have to train people up, and Microsoft doesn't offer any of the employment perks that they used to use to attract top talent (Seattle used to be a low CoL area, everyone had private offices, job stability).
When I started at Microsoft in 2007, the interview bar included deep knowledge of how computers worked. It wasn't unusual to have meetings drop down to talking about assembly code. Your first day after orientation was a bunch of computer parts and you were told to "figure out how to setup your box".
Antivirus wasn't mandatory. The logic was if you got a virus, they made a mistake hiring you and you deserved to be fired.
When your average developer can go that deep on any topic, you can generally leave engineers well enough alone and get good software.
> But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way.
It doesn't help that there are some teams that are hardcore in keeping things as they are and don't want any tooling that might improve COM development experience.
To this day Microsoft is yet to have any COM related tooling for C++ as easy to use as C++ Builder does it.
MFC, ATL, WRL, WIL,.... you name it.
The only time it seemed they finally got it, with C++/CX, there was a group that managed to kill this product, replace it with C++/WinRT, with no tooling other than the command line IDL compiler, now also abandoned as they refocused into windows-rs.
Oh, as somebody who wrote C++/CX code at the time, I was very pissed when they replaced it with WinRT.
1 reply →
Antivirus wasn’t mandatory in 2007 after the 2003 Blaster Worm, that required no user action to compromise the PC? Wild
On the other hand there was e.g. CVE-2021-1647 where Microsoft's antivirus would compromise the PC with no user action.
(At least I think that's the one I'm thinking of. It's marked as a high-severity RCE with no user interaction but they don't give any details. There was definitely at least one CVE where Windows Defender compromised the system by unsafely scanning files with excessive privileges.)
People forget that prior to Microsoft releasing Defender, antivirus on Windows was universally bad. Like "make your machine almost unusable" bad.
This was also before SSDs as well.
With local build times already measured in multiple hours (large C++ code bases, lots of caches obj files loaded from central build servers to make local incremental rebuilds even possible), Microsoft didn't want to make things worse by forcing any bloat on developer machines.
Maybe they fired everyone who was working there in 2003. Would explain some things.
“One team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.”
Jesus, you have tons of people who are willing to do that, even now. Microsoft just don’t care to hire from non-target schools, or ordinary professionals and train them —- sure the reason is, people believe that you cannot improve mediocrity, which I don’t believe so.
On a completely different page, most of the generals and advisors and high level bureaucrats of the first Emperor of the Han dynasty came from exactly one county — the county of Pei. But in peaceful time they are just “ordinary people”.
> Jesus, you have tons of people who are willing to do that, even now. Microsoft just don’t care to hire from non-target schools, or ordinary professionals and train them
Microsoft was never elitist about what schools they hired from. When I was there almost anyone who applied from an accredited CS program got at least a phone screen.
But no one in their right mind, in 2012 (when this particular incident happened!), would voluntarily pick up native Windows development skills. It was obvious even then that it was a dead end market.
The number of companies hiring native Windows developers is tiny, and the pay isn't all that good.
It isn't quite COBOL bad, but it isn't a growing market.
2 replies →
Yeah I thought that was extreme. An engineer going to the board of any corporation let alone Microsoft is not normal or customary IME. That could explain why they got no response.
When you see significant risks to the org and its value, and they go completely unaddressed by management, the board is the final step before going to the public. It is the board’s duty to the public owners to make sure management isn’t driving the company into the ground.
It would be interesting to see this raised in the next shareholders meeting as a question of whether the board and exec team are actually competent and doing their work.
A man can dream anyway. When there is this much money on the line, sometimes people actually get held somewhat accountable.
It's a baffling flaw in human nature. The board should have cared about these issues, but in practice communications to and from the board are tightly controlled, and communications outside of those constraints are discarded.
This occurs whether or not it makes sense. Machiavelli actually warns about the specifically: if someone else controls access to you and communication with you, they have real leverage over you.
Not on day one. Imagine it took two years to get there.
“customary” referred to the path through the Secretary, as opposed to writing directly to members. Besides that, depending on the nature of the communication, if everything fails, you may need to be sure you talk to people who will unconditionally put the best interests of the company ahead of any other consideration. The Board is one such group. See what Boeing did with the report of the mechanic who saw flaws in the 737 MAX’s door plugs. Was that worthy of a letter to the CEO, then the Board if no reaction? Or just talk to your dismissive manager and let the planes crash? I made a judgment call, which I entirely own.
The CEO is accountable to the board. If they are derelict in their obligations to the company, that's where you need to raise a stink so they can fix it.
Well, yeah, that’s what a board does, but I think the issue is whether it is customary to go to the board directly in this situation. The answer is a resounding NO. Very odd, but cool idea and approach.
Maybe naive, but why not? If it's a serious enough issue, and you're not getting anywhere through your management chain all the way up to the CEO, why is it novel to contact the people the CEO reports to? They're not royalty, they're other human beings who also eat, piss and fart like everyone else.
3 replies →
“customary” referred to the path through the Secretary, as opposed to writing to individual members. I added a clarification at the bottom of the page.
Yeah but I can't conceive a world where a Board would care about technical complaints from an employee about engineering decisions several levels downstream of the CEO's executive domain.
Yes it is that unreliable. Even when given free credits, I would rather pay for the offerings from Amazon/Google.
He is, I think, Swiss, perhaps a cultural difference?
We like things well done, but also integrity and accountability.
>> He is, I think, Swiss, perhaps a cultural difference?
> We like things well done, but also integrity and accountability.
Unless they involve secret bank accounts [1], refugees [2], and/or nazis [3] :-)
All props to you, though, for speaking out. This is going to help a lot of folks understand why things are going the way they are with Azure, and MS.
[1] https://www.theguardian.com/news/2022/feb/22/how-swiss-banki...
[2] though in fairness, it appears to be changing, but we shall see. https://www.swissinfo.ch/eng/culture/how-switzerland-s-views...
[3] https://www.theguardian.com/world/1999/dec/11/1
1 reply →
Azure is when you have a different version of the same product/api in each region.
I notice the title mentions the author is a former employee but he never mentions the terms on which he left.
at the bottom of part 4 -
> The org’s leadership responded with strong defensiveness and denial. Not long afterward, the organization terminated my employment.