Man Boeing sounds like such a disgusting place to work. Granted, I'm sure many, many large publicly-traded companies are like this. But the way the article lays it out these executive initiatives reminds me of mobster movies where they move in and milk an honest citizen's store for all it's worth.
That profits and executive bonuses should take precedence over the safety of human lives is horrible. I'm curious to hear more about why the unions formed, since it sounds like there was a very contentious relationship there.
People are bad at understanding risk. Tell someone that this change will save the company enough money to bump the share price by $2, but there's a one-in-a-million chance per flight that it will cause a crash that will destroy the company instead. Do you go for it? A lot of people will hear "one in a million" as "basically impossible" and say it sounds great. They'll probably rationalize it further by deciding that you're being paranoid about the risks. And then a while later you have a few hundred dead people and a company with a shattered reputation.
And to further complicate it, the smaller a risk, the worse we are at estimating it.
Investigations after the Challenger Disaster revealed that engineers at NASA thought the space shuttle had a 1/100 failure rate per mission, while managers thought the shuttle had a 1/10,000 failure rate.
From a completely technical perspective, one can understand the cost of this risk with real option valuation techniques. However I’d say this is an inappropriate method when so many lives are at stake.
by making unsafe aircraft Boeing was able to avoid losing customers, and in fact make many more sales because airline’s pilots didn’t need to be retrained and their equipment didn’t need to be changed. Combined that made the MAX a more cost effective alternative to the airbus.
That’s your shareholder value, as long as your unsafe aircraft don’t end up crashing.
Also it’s mostly nonsense: executive compensation is often tied up in revenue and market performance, and they’ve mostly cashed out on that before the financial damage caused by their “work”.
We need to stop using "profit motive", "shareholder value", and "rational markets" as if they somehow provide moral air cover for actions that cause easily avoidable deaths and/or massive damage to the commons.
Everything today is about shareholder value. Nobody, but nobody gives a flying frog's fat fucking ass about the end user/customer/employee/etc any more.
The simulators are certified training devices, which means that you can do part of your legally required training in them (and not have to fly an actual jet).
To be certified as a training device it has to be accredited to act similar to the real thing. That accreditation costs money.
That's not to say it couldn't be done cheaper, but until someone does $15 million is what the market will bear. The other option is gassing up a real jet for training.
Our industry needs to take the 'Engineer' part of Software Engineering a lot more seriously. A professional certification to use the title is looking more and more important with each incident like this, otherwise developers have no teeth to push back when flaws like this are brushed aside by management to save money/reduce costs/etc.. Cyber security, data ethics,.. many things in programming could benefit from certification, and having to legally sign off on your own work.
First off, this is nonsense: the software was doing exactly what it was meant to (and designed to) do. Hardware engineers chose not to provide multiple sensors to validate AoA, hardware engineers did not provide a human-capable override. MCAS was designed to not be disabled by pilots, because doing so would make the plane a different aircraft according to the FAA.
Anyway I have yet to see a software related “certificate” that isn’t rote-learnable, comically high level, or both.
You also have to ask, what are you certifying?
All of these are fairly trivial to avoid in small programs:
* Use after free
* time of check/time of use
* out of bounds
* numeric overflow
Especially in any kind of test environment where you are being extra careful.
Then there’s the language problem: many engineers have to use multiple languages, some only have to use “safe” languages. Should you require a different cert for each?
You’re also saying “not everyone gets to write software anymore” because the certification won’t be free.
How does open source then work? Clearly people working on the Linux kernel should be certified, so now you’re saying Linux should only accept patches from people who live in countries that can provide the required certs.
> Hardware engineers chose not to provide multiple sensors to validate AoA, hardware engineers did not provide a human-capable override. MCAS was designed to not be disabled by pilots, because doing so would make the plane a different aircraft according to the FAA.
I have two comments. One replace hardware engineers with 'management'
The second when I've read people talk about validating the AOC readings it makes me twitch a bit. Partly because my day job involves firmware that manages a self organizing sensors network. Validation of sensor data sounds easy until you force yourself to conceptualize what the system can know based on the actual data it sees and not your perceptions.
More there is a strong tendency to over focus on the ordinary case. And not all the edge cases. Very often dealing with edge cases is the fundamental problem. Consider designing the front end of a car. The primary design goal is actually 'passengers don't die when you drive it into a tree'
Problem with the MCAS system is it needs to work under all the edge cases, not just when the plane is flying in smooth air while the pilot is pulling the nose up. Like during a hard turn into wind-shear.
This isn't a certification problem. It's an engineering ethics problem.
The pass the buck circle jerk is how this design flaw came to exist. Everyone in the engineering organization needs to have the balls to point out systems design errors. Management needs to listen to them and not issue "make it work" marching orders. Regulators need to not delegate their responsibility to the previous.
More than one person could have put their foot down and demanded triple redundancy. That this didn't happen suggests even more safety concerns lurk in all of Boeing's products.
Currently, the avionics software is certified, not the software engineer. The FAA-delegate safety reviewers get special training, but otherwise a bachelor's degree in a related discipline is the standard for an individual contributor's formal education.
There is arduous process in place to help ensure that commercial avionics software is produced to an acceptable level of quality. Problems can still get through, but the process helps weed out a lot of issues that you'd likely see in non-safety-critical software.
this was a systems engineering failure. nothing more. the system is designed to find these and remove them. it has not been determined if this is because of cost cutting or management pressure. could be, but it is also possible it is just an error made by people.
> Hardware engineers chose not to provide multiple sensors to validate AoA.
In effect you've just shifted the blame. Developers working at the lower levels could've pushed back on this harder if they were legally required to. My point is if mechanical and electronic engineers are liable then so should software guys - they need more power to say no.
> You also have to ask, what are you certifying?
An argument could be made that formal verification & ethics would be useful in this context.
> You’re also saying “not everyone gets to write software anymore” because the certification won’t be free.
Degrees aren't free either. Most developers aren't working in aerospace and won't need the rigour.
> How does open source then work?
I'm not talking about OSS. I'm talking about people who work with software that can kill people. If the Linux kernel is used as a technology in these machines then the software 'engineer' who made that decision is legally liable. The blame stops with them.
MCAS was designed by aeronautical engineers, not software engineers. The exact sensors, function, and responses were all specified by aero engineers. The software, as far as we know, was produced using acceptable software engineering processes and functioned exactly as designed.
What, exactly, do you think a PE cert for a software engineer would have done here? Do you think the software people should have refused what the aero people certified as safe?
Require signing off on something as an Engineer that they understand the scope for which this unit is being utilized and that it is reasonably safe according to the best practices known at the time and their own full and complete understanding of the context.
It gives legal teeth for them to say; "No, this has not yet been proven to be safe, I cannot sign on to that". However at the same time a union or guild is required so that management doesn't penalize for being a moral engineer versus a rubber-stamp engineer.
Most software engineers don’t even have a college degree in computer science, and now we want them to get a professional certification? Good luck with that.
The title 'Software Engineer' is an over used term imo that describes any developer out there.
That title should be reserved for those that have the same credentials as an ME, EE, etc. Someone who is a CS degree holder or a self taught comp dev have in no way the same training as someone with a CE degree.
If anything you've just reinforced my point that there needs to be a higher barrier to entry when dealing with software that can critically affect human life.
certifications are nonsense I hear this everywhere I go any time I mention oh I'm thinking about doing XYZ certification it's like don't waste your time doing that just learn what you're talking about and prove it certifications have a lot of work around sort whatever anybody can study and pass a test doesn't mean you can do the work
Software certs have a bad name because of things like Oracle and Microsoft multiple choice tested certifications that are cheated outrageously, and crap like one-day certified ScrumMaster nonsense.
Probably the better model would be the apprentice/journeyman/master progression from the medieval trade guilds.
With this power, comes serious responsibility. If an engineer signs off on a design that they know to be fundamentally unsafe, that engineer has liability regardless of the internal pressures placed on them.
The incentives just aren’t there though. Performance reviews are all about impact, and engineers who focus on quality instead of impact are worse off in career advancement. Likewise with the incentives on companies; companies that do slow but careful development get overtaken by faster moving competitors which reward impactful employees.
And it isn’t even clear to me that most consumers would prioritize security / stability over feature-sets when choosing software.
How does a software engineer know that something is safe? Do they need to be aerospace engineers as well? Do they need to go over the full schematics of the hardware their software is running on?
Just throwing out there, the CFO is a gay gentleman with several "sugar babies". He hosts coke filled wild parties at his penthouse in San Jose. Might give an insight to the character of the person probably demanding all the cost cuts.
This seems like another case to stop H1B visa system. We are sacrificing quality over Q4 earnings by outsourcing skilled jobs to unskilled cheap body shops.
"Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents."
Uh what? H1-B’s are literally the opposite of outsourcing.
Yes there were outsourcing companies that abused the system (harming legitimate companies), but if you think outsourcing companies are going to magically start recruiting top of the barrel engineers you are sorely mistaken.
Man Boeing sounds like such a disgusting place to work. Granted, I'm sure many, many large publicly-traded companies are like this. But the way the article lays it out these executive initiatives reminds me of mobster movies where they move in and milk an honest citizen's store for all it's worth.
That profits and executive bonuses should take precedence over the safety of human lives is horrible. I'm curious to hear more about why the unions formed, since it sounds like there was a very contentious relationship there.
TFA mentions shareholder value. How exactly do unsafe planes help shareholder value?
People are bad at understanding risk. Tell someone that this change will save the company enough money to bump the share price by $2, but there's a one-in-a-million chance per flight that it will cause a crash that will destroy the company instead. Do you go for it? A lot of people will hear "one in a million" as "basically impossible" and say it sounds great. They'll probably rationalize it further by deciding that you're being paranoid about the risks. And then a while later you have a few hundred dead people and a company with a shattered reputation.
And to further complicate it, the smaller a risk, the worse we are at estimating it.
Investigations after the Challenger Disaster revealed that engineers at NASA thought the space shuttle had a 1/100 failure rate per mission, while managers thought the shuttle had a 1/10,000 failure rate.
From a completely technical perspective, one can understand the cost of this risk with real option valuation techniques. However I’d say this is an inappropriate method when so many lives are at stake.
The value is a gamble:
by making unsafe aircraft Boeing was able to avoid losing customers, and in fact make many more sales because airline’s pilots didn’t need to be retrained and their equipment didn’t need to be changed. Combined that made the MAX a more cost effective alternative to the airbus.
That’s your shareholder value, as long as your unsafe aircraft don’t end up crashing.
Also it’s mostly nonsense: executive compensation is often tied up in revenue and market performance, and they’ve mostly cashed out on that before the financial damage caused by their “work”.
We need to stop using "profit motive", "shareholder value", and "rational markets" as if they somehow provide moral air cover for actions that cause easily avoidable deaths and/or massive damage to the commons.
Everything today is about shareholder value. Nobody, but nobody gives a flying frog's fat fucking ass about the end user/customer/employee/etc any more.
Well, don't support public companies then if at all possible. Public companies = greedy shareholders
On the other hand, end users/customers/employee's don't care about schareholders either. Sounds fair to me.
1 reply →
Why do the simulators cost $15 million?
The simulators are certified training devices, which means that you can do part of your legally required training in them (and not have to fly an actual jet).
To be certified as a training device it has to be accredited to act similar to the real thing. That accreditation costs money.
That's not to say it couldn't be done cheaper, but until someone does $15 million is what the market will bear. The other option is gassing up a real jet for training.
But if the cost is discouraging the decision of what is legally required training, that is an issue.
Also, who accredits the simulator? The company that built it?
1 reply →
they cost more than that ! (and note they run the same code as the aircraft, in most cases)
Our industry needs to take the 'Engineer' part of Software Engineering a lot more seriously. A professional certification to use the title is looking more and more important with each incident like this, otherwise developers have no teeth to push back when flaws like this are brushed aside by management to save money/reduce costs/etc.. Cyber security, data ethics,.. many things in programming could benefit from certification, and having to legally sign off on your own work.
First off, this is nonsense: the software was doing exactly what it was meant to (and designed to) do. Hardware engineers chose not to provide multiple sensors to validate AoA, hardware engineers did not provide a human-capable override. MCAS was designed to not be disabled by pilots, because doing so would make the plane a different aircraft according to the FAA.
Anyway I have yet to see a software related “certificate” that isn’t rote-learnable, comically high level, or both.
You also have to ask, what are you certifying?
All of these are fairly trivial to avoid in small programs:
* Use after free * time of check/time of use * out of bounds * numeric overflow
Especially in any kind of test environment where you are being extra careful.
Then there’s the language problem: many engineers have to use multiple languages, some only have to use “safe” languages. Should you require a different cert for each?
You’re also saying “not everyone gets to write software anymore” because the certification won’t be free.
How does open source then work? Clearly people working on the Linux kernel should be certified, so now you’re saying Linux should only accept patches from people who live in countries that can provide the required certs.
> Hardware engineers chose not to provide multiple sensors to validate AoA, hardware engineers did not provide a human-capable override. MCAS was designed to not be disabled by pilots, because doing so would make the plane a different aircraft according to the FAA.
I have two comments. One replace hardware engineers with 'management'
The second when I've read people talk about validating the AOC readings it makes me twitch a bit. Partly because my day job involves firmware that manages a self organizing sensors network. Validation of sensor data sounds easy until you force yourself to conceptualize what the system can know based on the actual data it sees and not your perceptions.
More there is a strong tendency to over focus on the ordinary case. And not all the edge cases. Very often dealing with edge cases is the fundamental problem. Consider designing the front end of a car. The primary design goal is actually 'passengers don't die when you drive it into a tree'
Problem with the MCAS system is it needs to work under all the edge cases, not just when the plane is flying in smooth air while the pilot is pulling the nose up. Like during a hard turn into wind-shear.
5 replies →
This isn't a certification problem. It's an engineering ethics problem.
The pass the buck circle jerk is how this design flaw came to exist. Everyone in the engineering organization needs to have the balls to point out systems design errors. Management needs to listen to them and not issue "make it work" marching orders. Regulators need to not delegate their responsibility to the previous.
More than one person could have put their foot down and demanded triple redundancy. That this didn't happen suggests even more safety concerns lurk in all of Boeing's products.
You also have to ask, what are you certifying?
Currently, the avionics software is certified, not the software engineer. The FAA-delegate safety reviewers get special training, but otherwise a bachelor's degree in a related discipline is the standard for an individual contributor's formal education.
There is arduous process in place to help ensure that commercial avionics software is produced to an acceptable level of quality. Problems can still get through, but the process helps weed out a lot of issues that you'd likely see in non-safety-critical software.
2 replies →
Nobody is saying that you can't be a programmer, or a dev, you just can't be a software engineer without a certification.
9 replies →
this was a systems engineering failure. nothing more. the system is designed to find these and remove them. it has not been determined if this is because of cost cutting or management pressure. could be, but it is also possible it is just an error made by people.
Every 737 has multiple AoA sensors. (Reportedly each computer is only wired to one though).
1 reply →
> Hardware engineers chose not to provide multiple sensors to validate AoA.
In effect you've just shifted the blame. Developers working at the lower levels could've pushed back on this harder if they were legally required to. My point is if mechanical and electronic engineers are liable then so should software guys - they need more power to say no.
> You also have to ask, what are you certifying?
An argument could be made that formal verification & ethics would be useful in this context.
> You’re also saying “not everyone gets to write software anymore” because the certification won’t be free.
Degrees aren't free either. Most developers aren't working in aerospace and won't need the rigour.
> How does open source then work?
I'm not talking about OSS. I'm talking about people who work with software that can kill people. If the Linux kernel is used as a technology in these machines then the software 'engineer' who made that decision is legally liable. The blame stops with them.
4 replies →
MCAS was designed by aeronautical engineers, not software engineers. The exact sensors, function, and responses were all specified by aero engineers. The software, as far as we know, was produced using acceptable software engineering processes and functioned exactly as designed.
What, exactly, do you think a PE cert for a software engineer would have done here? Do you think the software people should have refused what the aero people certified as safe?
Require signing off on something as an Engineer that they understand the scope for which this unit is being utilized and that it is reasonably safe according to the best practices known at the time and their own full and complete understanding of the context.
It gives legal teeth for them to say; "No, this has not yet been proven to be safe, I cannot sign on to that". However at the same time a union or guild is required so that management doesn't penalize for being a moral engineer versus a rubber-stamp engineer.
3 replies →
Most software engineers don’t even have a college degree in computer science, and now we want them to get a professional certification? Good luck with that.
https://stackoverflow.blog/2016/10/07/do-developers-need-col...
The title 'Software Engineer' is an over used term imo that describes any developer out there.
That title should be reserved for those that have the same credentials as an ME, EE, etc. Someone who is a CS degree holder or a self taught comp dev have in no way the same training as someone with a CE degree.
Engineers are able to take their PE exam in either CE or SE. https://ncees.org/engineering/pe/
3 replies →
If anything you've just reinforced my point that there needs to be a higher barrier to entry when dealing with software that can critically affect human life.
3 replies →
certifications are nonsense I hear this everywhere I go any time I mention oh I'm thinking about doing XYZ certification it's like don't waste your time doing that just learn what you're talking about and prove it certifications have a lot of work around sort whatever anybody can study and pass a test doesn't mean you can do the work
Software certs have a bad name because of things like Oracle and Microsoft multiple choice tested certifications that are cheated outrageously, and crap like one-day certified ScrumMaster nonsense.
Probably the better model would be the apprentice/journeyman/master progression from the medieval trade guilds.
3 replies →
anti degree anti cert anti everything arbitrary $ barrier
With this power, comes serious responsibility. If an engineer signs off on a design that they know to be fundamentally unsafe, that engineer has liability regardless of the internal pressures placed on them.
The incentives just aren’t there though. Performance reviews are all about impact, and engineers who focus on quality instead of impact are worse off in career advancement. Likewise with the incentives on companies; companies that do slow but careful development get overtaken by faster moving competitors which reward impactful employees.
And it isn’t even clear to me that most consumers would prioritize security / stability over feature-sets when choosing software.
How does a software engineer know that something is safe? Do they need to be aerospace engineers as well? Do they need to go over the full schematics of the hardware their software is running on?
4 replies →
>>> If an engineer signs off on a design that they know to be fundamentally unsafe
The problem is you don't know its unsafe. It sometimes takes a disaster to shed lite on a problem. Engineering and design is hard.
Couldn't you make this argument about a mechanical engineer?
1 reply →
Now you can judge who is lying or who is telling the truth Just watch this and save yourself from others https://howto105.blogspot.com/2019/03/detect-lie1.html
Love between lion and man See in video how much they are taking care of each others http://bit.ly/2Yh5oBe
Very funny pets video of 2019 Just watch I am sure you can't control your laugh http://bit.ly/2E1UsQs
Love of elephant With human,see this video http://bit.ly/2Jg49Pp
That's sho cute video Dogs are dancing part2 Watch this on link below http://bit.ly/2W2DZpi
Just throwing out there, the CFO is a gay gentleman with several "sugar babies". He hosts coke filled wild parties at his penthouse in San Jose. Might give an insight to the character of the person probably demanding all the cost cuts.
This seems like another case to stop H1B visa system. We are sacrificing quality over Q4 earnings by outsourcing skilled jobs to unskilled cheap body shops.
That has more to do with outsourcing and haphazard disintegration of product requirements than H1B's.
"Eschew flamebait. Don't introduce flamewar topics unless you have something genuinely new to say. Avoid unrelated controversies and generic tangents."
https://news.ycombinator.com/newsguidelines.html
Uh what? H1-B’s are literally the opposite of outsourcing.
Yes there were outsourcing companies that abused the system (harming legitimate companies), but if you think outsourcing companies are going to magically start recruiting top of the barrel engineers you are sorely mistaken.
This is exactly what Agile succeeds in. This story is going to be an 8 point story? But Jim said he could get it done in 2.
Have a safety concern? The product owner doesn't think it's a priority.
What you describe sounds like Scrum done very poorly, not "Agile" in general.
>This story is going to be an 8 point story? But Jim said he could get it done in 2
If you treat individual ticket SP values as measures of time, you've already completely lost the point of everything.
Does Boeing use agile when deciding things like safety features on planes?
if not, is this relevant to the story?
Do they? I have no idea. I'm not an employee of Boeing.
Have I seen agile/scrum used to cut corners on and dismiss engineering concerns? Absolutely.