Comment by rossdavidh
1 day ago
So, a couple years ago Microsoft was the first large, public-facing software organization to make LLM-assisted coding a big part of their production. If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.
So, either LLM-assisted coding is not delivering the benefits some thought it would, or Microsoft, despite being an early investor in OpenAI, is not using it much internally on things that really matter to them (like Windows). Either way, I'm not impressed.
I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene). It started in 2014 and the trickle never stopped.
Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one, trying to maximize short-term shareholder value at the cost of long-term company reputation/growth. It is very common and typical of US Corporate culture today, and catastrophic in the long-run.
[0] https://arstechnica.com/information-technology/2014/08/how-m...
[1] https://www.reuters.com/article/business/microsoft-expected-...
That was in 2014, doesn't explain the timing of these increasingly common broken patches. I had never gotten as many calls over Windows Update messes from my non-techie family as last year.
The lack of QA isn't felt right away. They are accumulating tech debt, which mean problems are becoming more frequent and harder to solve over time until they fix the fundamentals, and it doesn't feel like they intend to.
5 replies →
Oh boy, in 2015 Windows 10 was released, and it was extremely broken, including endless reboot loops, vanishing start menu and icons, system freezes, app crashes, file explorer crashes, broken hardware encryption and many broken drivers – so really it was about the same as now. Embracing LLMs and vibe-coding all around made this even worse of course
14 replies →
Just because it’s getting worse faster doesn’t mean that it wasn’t getting worse before
The arstechnica article was very good as a history of waterfall v sprint using MS as a case study. However the firing the QA department narrative is not supported:
Prior to these cuts, Testing/QA staff was in some parts of the company outnumbering developers by about two to one. Afterward, the ratio was closer to one to one. As a precursor to these layoffs and the shifting roles of development and testing, the OSG renamed its test team to “Quality.”
Two QA per dev?? That seems ginormous to me. What am I missing about the narrative about evil corp sending all of QA packing, that seems not supported here?
The second, Reuters article seems like it's saying something different than the QA firing narrative - it seems to talk about Nokia acquisition specifically and a smattering of layoffs.
Not supporting layoffs or eliminating QA, and I'm deeply annoyed at Windows 11. I just don't see these as supportive of the narrative here that QA is kaput.
> Two QA per dev?? That seems ginormous to me. What am I missing about the narrative about evil corp sending all of QA packing, that seems not supported here?
I think you're underestimating the QA burden for large parts of the company. When I worked in payments at MS, the ratio of QA to dev after the cuts was probably on the order of dozens to one, if not a hundred or more once you threw in Xbox/Windows/etc accessibility QA from across the organization and all the other people like lawyers involved in handling over a hundred jurisdictions. I was little more than a frontend line cook and even I had three QA people reporting directly to me; two of them helping write tests so they ostensibly should have been automating themselves out of a job.
There is a lot of manual testing when you have a complex system like that where not everything can be properly stubbed out, emulated, or replaced with a test API key. They also have to be kept around to help with painful bursty periods (for us it was supporting PSD2, SCA, or 3DS2, forgot which). Payments is obviously an outlier because there is a lot of legal compliance, but the people I knew in Cloud/Windows also had lots of QA per dev.
I wouldn't be surprised if the degradation in feature parity of newer Windows software was a result of this loss of QA. Without the QA, the developers have to be less ambitious in what they implement in order to meet release schedules, and since they don't have experienced QA they can't modify the older codebases at all to extend them.
1 reply →
In writing life critical systems like the Space Shuttle's operating system, effectively 99.9% of all work is QA.
MS had the dominant operating system in the world, and keeping its userbase and its ~monopoly dividend would have been more profitable as a business than doing... everything it's done in the past twenty years. Selling software that all the people use all the time just has a lot less opportunity for growth than making new software, according to Investor Brain.
1 reply →
The Windows ecosystem is insanely complex. And they supported it, because of the focus on QA and testing the company adopted 20 years ago after the Blaster worm.
I have a few pretty awesome teams stuck managing windows. They find bugs all of the time. The process of fixing them now practically requires a detachment of druids and Stonehenge to track where in the windows/lunar/solar cycles we are and how to deal with the bullshit & roadblocks the support and product teams throw up. If you fall for their tricks, you’ll miss the feature window… no fix for 18 months.
It used to be much easier as a customer in ye olden times, and I never felt that the counterparty at Microsoft was miserable or getting punished for doing their jobs. We feel that now as customers. You didn’t establish relationships with engineers like with other vendors, but there was a different vibe.
The focus of the company moved in to Azure, service ops, etc.
I worked in the windows org around that time and the Dev/QA ratio there was closer to 1:1. QA did both manual testing and much of the automation, quality gates, and did regression testing against older versions of windows. Given the complexity of the product is is fairly easy for an inexpensive change to require an expensive test effort.
I had a QA engineer who gave me feedback on designs, great code reviews, and who wrote tests that I could also run.
It was a partnership. I miss it.
1 reply →
In the chip design world, 2:1 for design verification to design is on the low end of normal.
Some organizations have gone as low as 1:1 but that is considered an emergency that must be fixed. It’s so important that designers will be intentionally underworked if there are not enough validation engineers on staff.
When you can’t fix bugs in the field, quality is important.
> Two QA per dev??
QA is a lot cheaper than dev. If your goal is to make quality software* on a fixed budget, you want to be QA-heavy.
* Note: the OS definition of "quality software" drastically differs from your average app.
15 replies →
> Two QA per dev?? That seems ginormous to me.
The only person I heard was writing perfect code was Donald Knuth. And even he had bugs in its code.
> I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department
A move no doubt encouraged by c-suites to demonstrate how effective LLMs are in the budget tally.
There's a great talk that explains how code structure ends up looking like the org chart, and every subsequent organization chart layered on top producing spaghetti code. Windows is now old and full of spaghetti code. Then Microsoft layed off all the expensive seniors who knew the stack and replaced them with cheaper diverse and outsourced staff. Then the people who can't maintain the code use AI and just ship it without any testing.
edit: https://en.wikipedia.org/wiki/Conway%27s_law
According to Microsoft's top brass, Copilot (one of them) should easily be able to handle QA. So OP's point remains.
The shift from an engineer-led corporation to an MBA-led corporation has brought Boeing close to the brink of collapse.
There seems to be a lot of internal factionalism that's showing up in the final product. I think this is a chromic disease that flares up every couple of years and is then clamped down on... but for whatever reason the lessons are never learned for long.
It has been an MBA company for most of its life. If I had to draw the line, IMO seems Windows 2000 was the last engineer-driven product, and by then it had already developed predatory habits.
There's always Windows Server...
Let's hope for the catastrophic scenario. A world without Microsoft.. no telemetry or backdoors. Please continue on this track to disaster!
Accelerationists seem to think the world after a vacuum is going to be some utopia
I think more competition is better than less
3 replies →
It's not only MS with an interest in maintaining these misfeatures in consumer tech. It's not even only private industry.
Indeed! I'll wait on the penguin or fruit side with some pop-corn and see where the things are going to.
3 replies →
>A world without Microsoft.. no telemetry or backdoors.
thank god microsoft is the only entity on the planet that uses telemetry or violates privacy. get rid of them and we're in a new age!
> but this is much more to do with Microsoft very publically firing the QA department[0][1] as a cost savings measure and claiming developers will do their own QA (long before LLMs were on the scene).
I will never ever understand this. Development and QA are two different mindsets. You _can_ do both, but you* can't be great at both.
* There's always exceptions, yes, yes.
I think all companies eventually mutate into a MBA company. For MSFT there was a culture from very early that PMs should lead the project instead of engineers. I read in "Showstoppers" that Cutler was very against of the idea and he pushed back. So that means even in the late 80s MSFT was already a MBA-centered company. The only reason that it has not degraded yet, was because it has not achieved the monopoly position. Once it does it started to chew on its success and quickly degraded into a quasi-feudal economic entity.
> Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one
I don’t think this is just Microsoft. Few engineers and visionaries that started these big companies are still at the helm.
It’s an opportunity for other companies to take over imo.
> It’s an opportunity for other companies to take over imo.
This is a feeling commonly shared here.
I'd like to point out that IBM still dominates the large, billion-dollars worth mainframe market, almost 70 years after it invented it, despite continuous mismanagement for probably 40 years.
Microsoft dominates the PC market 40 years after taking it over with MS-DOS, and despite multiple debacles (Windows Millennium, Windows Vista, now Win 11, probably others I'm forgetting).
Microsoft dominates the office suite market 30+ years after taking it over with MS Office, despite some huge controversies (the Ribbon still annoys nerds, to this day). More than that, Microsoft has leverage MS Office to become the close second cloud provider after AWS despite starting far behind it.
Google and Apple will probably dominate the smartphone and tablet markets for a long time, after taking over those markets 10+ years ago.
The market can stay irrational longer than you can stay solvent and a company with a massive moat can outlive most of us. I'd actually turn this on its head by saying that assuming a new comer will topple the incumbent "any day now" is the irrational approach to a market.
2 replies →
At least we get Visual Studio Code for free
Yeah they baited everyone, blocked Python and C# plugins and then closed down their marketplace to 3rd party editors. Classic EEE tactic.
Some useful tech has come out of the development of VS Code that every other editor has been able to benefit from but I don’t rate it much as an editor any more.
It’s rare for MS to do just the embrace and extend part of EEE, unless Copilot is the latent implementation of ‘extinguish’.
1 reply →
So essentially, they need to turn quality around or suffer the thousand cuts of death like Intel?
Although. These companies don't "die" - it's more the consumers end up being abandoned in favour of B2B?
> I know blaming everything on LLMs is in vogue right now; but this is much more to do with Microsoft very publically firing the QA department.
Yes, yes, "agile" everything...
I remember clicking on a perfectly honest button in Azure Dev Ops (Production) and it told me that the button is completed but the actual functionality will be probably delivered in Sprint XY.
No one is blaming LLMs.
Their presence in this situation casts a conspicuous shadow though.
On a contrary note: if LLMs really are that helpful why are QA teams needed? Wouldn't the LLM magically write the best code?
Since LLMs have been shoved down everyone's work schedule, we're seeing more frequent outages. In 2025 2 azure outage. Then aws outage. Last week 2 snowflake outages.
Either LLMs are not the panacea that they're marketed to be or something is deeply wrong in the industry
Why not both? It's not this industry, it's everything. Fuck Jack Welch, fuck the Chicago School.
1 reply →
Wholeheartedly agree.
I can't wait until we can live in a better era where we look back with collective disgust at the blatant white-collar crime time period that was ushered by Friedman and Welch.
That, plus the current era, feels to me like a massive dog whistle for people who can't read satirical stories like A Modest Proposal without taking them as instructions.
Microsoft fired their QA because at the end of the day, they are beholden to shareholders. And those shareholders want higher profits. And if you want higher profits, you cut costs.
It's not a culture problem. It's a 'being a business' problem, which unfortunately affects all publicly-traded companies.
Shareholders are, on average, not this activist. A CEO can in fact run a public company with a long-term outlook instead of pumping the numbers for just the next quarter.
1 reply →
That’s a cop-out though. Company boards are legally required to act in the best interests of shareholders, and plenty of shareholders would agree that running a business in a sustainable way that can deliver profits over the long term is more in their interests than a business trading its future for some short term profits.
It’s a cultural problem really, where too many people who study business and economics have been taught this idea that it’s a moral necessity that businesses maximise profit for shareholders (to the point where plenty of people even wrongly believe that’s a legal requirement!), but it’s an ideological position that has only caused once great companies to fail and huge damage to our economies.
Are businesses expected to boom and bust? Cost cutting is fine if you don't kill the company in the process.
3 replies →
Microsoft has a cultural problem; it went from an "engineers" company to an MBA directed one
Every simplistic analysis of failing company X uses a hackneyed cliche like this. But in the case of MS, this is completely ridiculous. MS has been renowned for shitty software, since day one. Bill Gates won the 90s software battle based on monopoly, connections and "first feature to market" tactics.
If anything, the heyday of MS quality was the mid 2000s, where it was occasionally lauded for producing good things. But it was never an engineers company (that's Boeing or whoever).
The argument I usually hear is that you only truly get the 10x improvements with <new-model> (right now, Opus 4.5), so they've only had a few months, not years. In a few months, it'll turn out that <new-model> wasn't actually capable of that, but <new-new-model> is, and as that's not been out long its unfair to judge so early. And so the cycle begins anew
Very true. Look at the wikipedia article for "AI Winter" to see how old this cycle truly is...
I fully believe highly skilled people can get a great benefit from LLM tools; probably not 10x; but enough that its noticeable.
The key thing for me is that it only works when the LLM is used for tasks below the devs skill level; It can speed up somebody good, but it also makes the output of low-skill devs much harder to deal with. The issues are more subtle, the volume is greater, and there is no human reasoning chain to follow when debugging.
So you combine that with a company that has staff in low skill regions, and uses outsourcing, and while there might be some high skill teams that got a speed up, the org is structured in a way that its irrelevant.
I think they keyword is "highly skilled." However, not everyone using the LLM will be highly skilled, especially juniors new to the industry.
They weren't great before LLMs either.
Also, it seems from the outside like a dysfunctional organisation, or at least with incentives heavily misaligned with their users. Replace LLMs with a bunch of 10x engineers and it will still be bad in an environment like this.
So not sure how much to blame the LLMs - or in fact how much MS is really using them. Poor souls have to use MS AI tools, I almost feel sorry for them.
They hit peak with Windows 7 and will never have an operating system that good again.
Some flavors of Linux are approaching the Windows 7 peak as well as far as ease of use for newbies, software "just working", and for familiarity for users of other OS's.
Their days as the default OS for most people are numbered unless they pull an incredible heel turn.
On a whim I gave my 14 year old an old System76 laptop with ElementaryOS on it then sent her back to her Mom's house on the other end of the world. Then she switched schools and ended up requiring a laptop instead of an iPad to do her work. I about crapped my pants but she's been using that laptop almost problem-free for two months now (two glitches with Firefox that she got around). She even figured out how to install Sober so she can play Roblox. While that probably says as much about my parenting as Linux's progress I have to say, I'm pretty impressed.
Have you tried Windows 11? The WSL2 integration works really well. And the work that is being done in regards to safe vms so games can move away from kernel anticheat is also exciting.
1 reply →
They still need to land on consumer PC shops for regular users to take notice, until the trend of online only, and zero OEM support, rather reverse engineering even when there are systems out there like Dell XPS, Windows and macOS will keep being what most regular users buy.
Either that or a mix of tablets with detachable keyboards or Chromebooks, none of them GNU/Linux powered.
Oh it did help.
Microsoft went all in on do more with less and fired/reorged significant part of the company.
Wouldn’t be surprised if the outage is caused by new team taking something over with near zero documentation while all the tribal knowledge was torched away
Or LLMs weren’t good enough yet years ago, but the growth curve looked so promising that an investment seemed a good idea.
Also: do you have a reference for “a couple years ago Microsoft [made] LLM-assisted coding a big part of their production”?
I know they started investing, mentioning future benefits, but don’t remember them saying their Windows development team (heavily) relying on it.
I think it's naive to believe AI is used primarily for productivity boost. It's used mainly for cost reduction and to increase profits, even if quality and productivity take a hit in the process.
exactly my thoughts as well - if LLM really were massive productive booms - then we would see the number of bugs in major software platforms going down - we would see more features - but neither is happening
so yeah we're being sold a bag of air
It's not LLMs. It's returns-driven-development.
Growth at any cost. Once growth is unable to increase the wealth of the shareholders the money has to be diverted from elsewhere, via cuts. Money gotta keep flowing upwards.
But the second was always the case, windows and everything else is getting shittier so fast it would require a prompt explanation if we didn't have one.
Rose tinted glasses.
Windows 2000 may have been bearable but windows has always been shitty.
Microsoft is not even using dotnet core and what not, internally. SLT is very hard on adopting AI, but not much on getting results
If they used copilot and it was years ago, I'm actually impressed there are no reports of Windows PC's exploding
Everyone should read "The Bear Case for AI" thread:
"The bear case for AI is that bringing 10x or 100x or 1000x more intelligence to America will not change anything because U.S. institutions are already designed to ignore or waste intelligence and have no idea what to do with any more of it."
https://twitter.com/mmjukic/status/2014255931215716545
this reasoning is flawed.
wouldn't a for-profit company just balance the workforce for the productivity gained to increase overall profit?
some person is 10x 'more productive' (whatever that means) , let's cut 9 jobs.
Although to your grander point, employment during the LLM-embrace period seems fairly stable.[0]
[0]: https://stockanalysis.com/stocks/msft/employees/
(1) a couple of years ago, LLMs for coding sucked pretty bad.
(2) LLMs are a force multiplier. If you start with a negative number, then your coefficient makes things worse.
(3) Microsoft has never been a place of quality. It's not organized for that, it doesn't have that as its philosophy, and so you should never be surprised that it doesn't deliver that.
If anything, we see a decrease, not an increase.
Imagine a world where Microsoft was pushing “Copilot” integration everywhere, just as they are in this one—but the proof was, actually, in the pudding. Windows was categorically improving, without regression, with each subsequent update. Long-standing frustrations with the operating system experience were gradually being ironed out. Parts of the system that were slow, frustrating, convoluted, or all three, were being thoughtfully redesigned without breaking backwards compatibility, and we were watching this all unfold in real time, in awe of the power of “AI”, eyes wide with hope for the future of software, and computing in general.
Think of how dramatically this hypothetical alternate reality differs from the one we live in, and then consider just how galling it is that these people have the nerve to piss on our leg and then tell us it's raining. Things are not getting better. This supposedly-magical new technology isn't observably improving things where it matters most—rather, it's demonstrably hastening the decline of the baseline day-to-day software that we depend upon.
The distance between the promise and the reality really is huge. On some level I wished they'd just promise less, because it's not like LLMs compleatly useless. I don't find much use in them, but some clearly do. They do them. But since the entire economy has apparently bet the farm on AI, underpromising isn't really an option, while underdelivering is a problem for future Microslop and co.
Interesting thought experiment. In that alternate reality, their shareholders would probably be shouting "why would you give competitors access to this awesome tool?!"
I guess you haven't tried ZZK-5.6 with Maverick Agent? What prompt did you use? If it doesn't work, you can always try a swarm of agents with model hot-reload and re-spin. That will solve all your problems, write all your code and then make you a cup of coffee.
But web people can write css faster so I think it is a net positive?
Yeah, but I'm very worried about subtle errors getting introduced
[dead]
> If LLM's really delivered 10x productivity improvements, as claimed by some, then we should by now be seeing an explosion of productivity out of Microsoft. It's been a couple years, so if it really helps then we should see it by now.
That productivity may not be visible. I think MS's move-everything-to-rust initiate would be one hell of an endorsement if they manage to make visible progress on that in the next couple of years.
> That productivity may not be visible.
I'm not sure what your take is, but this reads like goalpost shifting.
If one of the biggest orgs that practically mandates some amount of LLM use cannot surface productivity gains from them after using them for several years, then that speaks volumes.
Reality has a way of showing itself eventually.
Microsoft has no "move-everything-to-Rust initiative" and never did. That was a bunch of clickbait created based on the personal comments by a single Microsoft developer.
Thanks for the heads up, I was not following closely.