Comment by ghoblin
9 days ago
There are only so many safety-first companies and products. The vast majority of the economy isn't optimizing for safety
9 days ago
There are only so many safety-first companies and products. The vast majority of the economy isn't optimizing for safety
Could it be that the only large safety-first companies are the ones forced by law (either proactively, or due to reliable legal accountability if things go wrong) to be safety-first?
> There are only so many safety-first companies and products
There are only so many companies that think of themselves as safety-first. In practice, basically all companies work on things that should be safety-first.
Does your software store user data? Congrats, you are now on the hook for GDPR and a bunch of similar data handling regulations.
Does your software include a messaging component? You are now responsible for moderating abusive actors in your chat.
Does your software allow users to upload images? Now you are a potential distribution vector for CSAM.
And so on... safety isn't just for things which can cause immediate death and dismemberment
There’s a difference between "safety matters" and “safety is the primary constraint". Most companies manage risk to an acceptable level while optimizing for speed and cost. Aerospace companies optimize for minimizing catastrophic failure, even at extreme expense. Treating a potential GDPR fine as equivalent to a flight-control failure ignores that society, regulators, and markets treat those risks very differently. The inconvenience and economic cost of your Discord messages leaking is not the same category of harm as your pacemaker controller failing. And because the majority of economic activity sits in that lower-criticality category, it would not be surprising if highly specialized, safety-critical human software engineering becomes more of a niche, while much of routine software development becomes increasingly automated or commoditized.
> Treating a potential GDPR fine as equivalent to a flight-control failure ignores that society, regulators, and markets treat those risks very differently
Agreed, though I think that if GDPR fines were actually being levied at the recommended 4% of global revenue, we'd start treating them more similarly to a 737 crash.
> The inconvenience and economic cost of your Discord messages leaking is not the same category of harm as your pacemaker controller failing
Sort of depends who they leak to. Your teen classmates who bully you to suicide? Your abusive ex who is trying to track you down to kill you? The 3-letter agency who is trying to rendition your family to an internment camp?
There are a lot of seemingly benign failure modes that become extremely lethal given the right circumstances. And because we acknowledge the potential lethality of something like a pacemaker failure, we have massive infrastructure dedicated to their mitigation (EMT teams, emergency external pacemakers, surgical teams who can rapidly place new leads, etc). For things society judges less important, mitigations are often few and far between