I wish this were a recent development, connected to major improperly reviewed code changes provided by LLMs, but let us be honest, MSFT has had an appalling, frankly embarrassing track record in this regard dating back literally a decade plus now.
I've experienced it more than once on my Surface back in the day [0], the entire globe was affected by Crowdstrike which also was caused by a lack of testing on MSFTs part and there are numerous other examples of crashes, boot loops and BSODs caused by changes they made throughout the years [1].
Frankly, simply, no matter whether the code changes are provided by the worst LLM or the most skilled human experts, it appears their review process has been faulty for a long time. Bad code making it into updates is not the fault of any new tools, nor (in the past) of unqualified developers since, frankly and simply, the review process should have caught all of these.
Mac OS can be buggy and occasionally is a bit annoying in my case (Tahoe though is actually rather stable besides a few visual glitches for me, surprising considering a lot of my peers are having more issues with it over 25) but I have yet to see it fail to boot solely due to an update.
Linux distros like Silverblue have never been broken due to an update in my experience (though there are famous examples like what happened a while back with PopOS). With immutable distros like Silverblue, even if you intentionally brick the install (or an update does break it), you just select the OSTree prior to the change and resolve any issue instantly.
For an OS one is supposed to pay for both with money and by looking at ads, Windows has been in an inexcusable state long before LLMs were a thing. Considering such major, obvious issues as "system doesn't start anymore" have been falling through code review for over a decade now, imagine what else has fallen through the cracks...
It isn't and I am apparently suffering from some very early onset dementia, so thanks for correcting me.
I, for some inexplicable reason, totally forgot that the whole Crowdstrike debacle was so bad because they could directly distribute faulty code to running systems, bypassing MSFT, staggered roll outs, etc.
I, again total mistake on my part, somehow had the mistaken memory that the changes were distributed via Windows Update, when the opposite being the case was what made that so bad.
Basically, mea culpa, honestly simple error and thanks for calling it out.
Having done contract development work for a number of different-sized software companies, a common rule I've noticed is the quality of the product is directly proportional to how many QA staff are employed. Clients that had me in direct contact with their QA teams provided high-quality bug reports, consistent reproduction steps, and verification of fixes that I could trust. Clients that did not have a QA team, where I was working directly with developers, usually had extremely fraught bug/fix/test cycles, low quality reproduction steps, fix validation that turned out to be not actually validated.
It's difficult for companies, especially big ones, because QA seems like purely a cost. The benefits are not obvious, so they're easy to cut when lean times come. But having people dedicated to the role of Assuring Quality actually really does accomplish that. If you are not delivering quality software, you are going to destroy user trust and lose to competitors. If the company is cutting QA staff disproportionately, that's a sign the leaders don't know what they're doing, and you should be looking for the exit (both as an employee & as a user).
I don't know what the right number of QA staff is, but it's probably higher than you think. At a small company I worked at previously, it was about 1 QA staff per 4 developers. That felt all right, but I certainly would have been happy to have more QA staff available to validate my work more quickly.
I wish this were a recent development, connected to major improperly reviewed code changes provided by LLMs, but let us be honest, MSFT has had an appalling, frankly embarrassing track record in this regard dating back literally a decade plus now.
I've experienced it more than once on my Surface back in the day [0], the entire globe was affected by Crowdstrike which also was caused by a lack of testing on MSFTs part and there are numerous other examples of crashes, boot loops and BSODs caused by changes they made throughout the years [1].
Frankly, simply, no matter whether the code changes are provided by the worst LLM or the most skilled human experts, it appears their review process has been faulty for a long time. Bad code making it into updates is not the fault of any new tools, nor (in the past) of unqualified developers since, frankly and simply, the review process should have caught all of these.
Mac OS can be buggy and occasionally is a bit annoying in my case (Tahoe though is actually rather stable besides a few visual glitches for me, surprising considering a lot of my peers are having more issues with it over 25) but I have yet to see it fail to boot solely due to an update.
Linux distros like Silverblue have never been broken due to an update in my experience (though there are famous examples like what happened a while back with PopOS). With immutable distros like Silverblue, even if you intentionally brick the install (or an update does break it), you just select the OSTree prior to the change and resolve any issue instantly.
For an OS one is supposed to pay for both with money and by looking at ads, Windows has been in an inexcusable state long before LLMs were a thing. Considering such major, obvious issues as "system doesn't start anymore" have been falling through code review for over a decade now, imagine what else has fallen through the cracks...
[0] https://www.computerworld.com/article/1649940/microsoft-reca...
[1] https://support.microsoft.com/en-us/topic/you-receive-an-eve... and https://www.eweek.com/security/microsoft-yanks-windows-updat... and https://www.404techsupport.com/2015/03/12/kb3033929-may-caus... and https://learn.microsoft.com/en-us/troubleshoot/windows-clien...
How was the Crowdstrike outage caused by a lack of testing on MS’s part?
(FWIW, Crowdstrike has also crashed Linux systems: https://lists.debian.org/debian-kernel/2024/04/msg00202.html)
It isn't and I am apparently suffering from some very early onset dementia, so thanks for correcting me.
I, for some inexplicable reason, totally forgot that the whole Crowdstrike debacle was so bad because they could directly distribute faulty code to running systems, bypassing MSFT, staggered roll outs, etc.
I, again total mistake on my part, somehow had the mistaken memory that the changes were distributed via Windows Update, when the opposite being the case was what made that so bad.
Basically, mea culpa, honestly simple error and thanks for calling it out.
> MSFT has had an appalling, frankly embarrassing track record in this regard dating back literally a decade plus now.
IMO, it's all traceable to their decision to lay off their dedicated QA teams in 2014
Having done contract development work for a number of different-sized software companies, a common rule I've noticed is the quality of the product is directly proportional to how many QA staff are employed. Clients that had me in direct contact with their QA teams provided high-quality bug reports, consistent reproduction steps, and verification of fixes that I could trust. Clients that did not have a QA team, where I was working directly with developers, usually had extremely fraught bug/fix/test cycles, low quality reproduction steps, fix validation that turned out to be not actually validated.
It's difficult for companies, especially big ones, because QA seems like purely a cost. The benefits are not obvious, so they're easy to cut when lean times come. But having people dedicated to the role of Assuring Quality actually really does accomplish that. If you are not delivering quality software, you are going to destroy user trust and lose to competitors. If the company is cutting QA staff disproportionately, that's a sign the leaders don't know what they're doing, and you should be looking for the exit (both as an employee & as a user).
I don't know what the right number of QA staff is, but it's probably higher than you think. At a small company I worked at previously, it was about 1 QA staff per 4 developers. That felt all right, but I certainly would have been happy to have more QA staff available to validate my work more quickly.
Everyone knows Microsoft’s pre-2014 OSes were oases of stability after all.
1 reply →