Comment by bargl
1 day ago
I worked on modernization in the FAA specifically around digitizing NOTAMs.
I would urge you not to consider this modernization from the perspective you may be familiar, as someone who may work on the bleeding edge and sometimes sees a lack of caution from engineers pushing for change.
The FAAs safety risk management is built around a rigorous process, but that is also a massive slowdown to reasonable progress.
One of the eng I worked with was 80+ years old and just didnt want to retire. He did not understand modern systems at all and the risks we were looking at were not (imo) realistically balanced.
I get the idea that the current administration is pushing safety to the side, but there is another risk where safety becomes a blocker to reasonable change.
When I left that work (more than 15 years ago) it was clearly a blocker.
This isn't like COBOL. Floppy disc were never reliable. They should've been phased out a long time ago
Magnetic degradation, wear from read/write heads, and environmental sensitivity mean disks have a natural expiration date and it's not long. And they are also easy to corrupt from all kinds of magnetic sources around us.
I get other stuff, even DOS like system. Safety critical software/hardware that was battle tested makes sense. Not floppy. Sometime it's just bureaucracy.
My floppies outlived many many models of usb thumb drives. I can still read floppies I recorded 30 years ago.
And I went through dozens looking for one that worked. Ymmv.
2 replies →
My Commodore Amiga 500 floppies from the late 1980ies still work. I went nostalgic a couple months ago, booted about eight of those floppies and all worked. They were probably kept dry the hole time, but I did move several times, they were in boxes in cellars and in garages.
> were never reliable
They absolutely mog the cancerous garbage that is NAND flash memory in terms of data retention. The fact that you can pick an 1980s floppy from the junk room and read it is telling. Meanwhile your average TLC SSD will probably get wiped in 3 years if kept in the drawer, and a QLC will get wiped in 1 year.
Reliability wise floppies are also better: flash memory leaks electrons naturally and gets shredded by radiation events, with temperaure accelerating these processes. Meanwhile magnetic disks are not altered by any normal magnetic field you could produce at home.
Survivorship bias. Floppies failed all the time. Flash can be unreliable if you’re talking about the cheapest or abused the wrong way, but flash SSDs handled wisely are much better than hdds , which in turn are much better than floppies.
If you compare the thumb stick that came free from a convention then compare it to a floppy that came free with a magazine. And in the end neither matters because they’re both cheapest possible junk, not fair examples of the tech.
Similarly, the cheapest led bulbs (put in poorly ventilated fittings) have crappy lifetimes, but that’s not the techs fault, it’s because they weren’t engineered to last, they are cheap crap. Well designed leds placed sensibly are vastly better than bulbs, but that’s hidden by the cheap end of the market
That is the first I ever saw the term "mog" used outside of bodybuiy/looksmaxxing.
More info here: https://www.urbandictionary.com/define.php?term=mog
This is just flatly false. You're comparing a worst case SSD with maybe 10% of its writes left to an extremely lucky floppy disk that hasn't died to binder hydrolysis, mold and fungal growth, warping or demagnetization. And you also completely made up that you couldn't generate a strong enough magnetic field at home to corrupt one - you absolutely could especially since rare earth magnets are not that uncommon and are more than enough to kill a floppy.
"mog"?
1 reply →
There's flash, and then there's flash.
Say, the Siemens PLCs I earn my living programming uses SD cards which range from 64kB to, uh, 128MB, methinks.
They still work after decades, so clearly, you can get long term reliable flash storage; just don't expect to be able to store your 4k holiday videos on them.
NOTAMs, on the other hand? Go ahead!
my 1980s 5.25" disks still work. i'm surprised but they have laster a long time.
I used to do floppy disk data recovery as a service for students at college. Roughly speaking, we helped one student out every three hours, out of 2,500 students, during busy hours. Based on the rates and hours, I think it would come out to something like 25% of the student body, each year, coming to us with a data loss incident involving floppy disks.
YMMV. I’m sure a lot of people were shoving these things into backpacks, and this was post-2000, so the quality of disks for sale was poorer than it was in the 90s and 80s.
I’ve also (since then, for myself) repaired drives. The drives fail because they’re mechanical and have motors, and anything that moves or has a motor in it will need some kind of service. Servicing a floppy drive requires specialized equipment which is hard to find (or requires a lot of patience—it’s possible to repair a floppy drive with just a screwdriver, but it’s not fun; guess how I know?)
3 replies →
3.5" floppies were still common while I was in college. I have distinct memories of copying a project to a floppy in my dorm room, popping the disk out, walking next door (literally next door in the dorms, not even going outside up or down stairs or anything), popping the disk into a team member's PC and the project being gone. We laughed it off and chucked the disk in the trash and got another. Project wasn't lost-lost since I had a copy on my hard disk. This was also the era of the 100MB Zip Drives, which I also had in my PC and thankfully all of our lab PCs had, too. That's what I normally used for all of my day-to-day file transfers of any real import. I still have my zip disks, but I seem to have lost my zip drive, so no idea if the disks still have any data on them.
13 replies →
Sure, but what are the odds they're using these rather than HD 3.5" disks purchased from whatever new old stock they've been able to source over the past 20-odd years?
and how many bits have rotted on those?
2 replies →
> disks have a natural expiration date and it's not long
You really have no idea what you are talking about. It's normal for floppies, especially 3.5" to last decades just sitting on a shelf. I've got 5.25" floppies from 1984, and they boot just fine on my old apple //e.
42 years in my parent's dank midwest basement.
Of course some will last longer. It's not a cliff, just "not recommended to use for longer than...". But a lot depends on the conditions of storage, how much usage they got, etc. Manufacturers gave them 10-20 years normally, which puts almost all of them in the not recommended territory.
The fact people have old working floppies doesn't change the fact that degradation happens.
Why do you write it as “Apple forward slash forward slash E”?
9 replies →
I remember losing half the light cues for a show because we tried to reload from the 3.5" floppy and it was corrupted.
Your writing like checksums aren’t existing. They exist!
I’ve bought some 15 year old floppies recently. They work. Had just to learn, that formatting them is required. In handling they are like SD-Cards.
Which is far better than the CDs, DVDs, Bluerays or records (you now…this big black discs with a needle).
And you don’t have to worry about network attackers. And we checksum everything from the network. For good reasons.
The more expensive blurays are quite long lasting, being non-dye based.
Absent doing something wildly abusive, vinyl records will play fine for a long time. And the degradation is comparatively graceful.
Source: go to any used record shop and find a copy of Chicago >5[0]. It'll almost certainly play, even if the music wasn't particularly good to start with.
[0] There are many copies of later Chicago albums in used record stores.
I'd learned long ago to treat floppies like Mission Impossible tape reels, and just assume the data would self-destruct in about 5 seconds.
And yet, there’s a reason we use magnetic tape for long term archive and it’s not because usb drives are reliable.
Without having details on the specific floppy disks in question there’s no way you can authoritatively say they are unreliable. They absolutely produced floppy disks that were far more reliable than the ones sourced from the cheapest vendor possible some of us used in school back in the day. Ignoring the fact I’m guessing the disks in question aren’t having to survive life in a students backpack.
Ohhh, don't compare magnetic backup tapes to floppy. Same designs underneath but completely different storage shell, delivery mechanism, even the coating is different.
One cost $100 another $1.
Then these tapes are typically write once read many.
They were designed to deal with floppy disk flaws consumers deal with.
Actually really cool to read about modern tape systems.
I had a customer with a slightly dusty office and we simply gave up on tape drives as a backup storage medium because we were replacing the drives or tapes on a monthly basis.
We switched them over to external USB enclosures with cheap SATA drives in them and never looked back.
> And yet, there’s a reason we use magnetic tape for long term archive
Is it because "long-term archival" strategies aren't updated frequently?
2 replies →
What would you have suggested in the 1980s instead of a floppy disk? And today?
If you have enough floppy disks they are a perfectly reliable solution.
50% of the time it works every time!
And yet they’ll replace it with some cloud-native monstrosity that will get hacked by an unhinged GPT.
Say what you will about floppies. There’s something to be said for a system that’s air-gapped by default.
>The FAAs safety risk management is built around a rigorous process, but that is also a massive slowdown to reasonable progress.
If this is the case, why was an airport knocked offline due to a cut fiber optic? A safety focused system would have a backup system say satellite or mobile 4G/5G. But they didn't have such.
This isn't about safety. Its about bureaucracy wrapped up as safety.
The primary was being inspected and the secondary failed.
There were two independent systems.
Which is why many systems on most airliners are redundant in threes (though under double fault may operate in a degraded mode).
9 replies →
> The primary was being inspected
So in reality you didn't have two systems.
otherwise you would have shut down operations until you had another connection live, so you had 2 online connections.
Ok but downtime isn't the same as failure. Downtime means planes can't take off. Failure means people die.
An more important question is what do they plan to replace it with?
>If this is the case, why was an airport knocked offline due to a cut fiber optic? A safety focused system would have a backup system say satellite or mobile 4G/5G. But they didn't have such.
There is a strong difference between an airport going down and ATC causing and accident.
Uptime isn't safety, they have very orthogonal characteristics.
Ridiculous false dichotomy. The bureaucracy can get in the way sometimes, but ultimately it is about safety, and the FAA has done an amazing job over the past several decades regulating the skies and played a massive role in making aviation as safe as it is. Standards are rigid and very well defined and change slowly because aviation is an industry where mistakes create massive disasters. There is some collateral inefficiency from things moving slow but thats way preferable to the collateral damage that would be caused by "move fast break things".
Although you were generally right, you can't say that after the parties between Boeing and FAA agents, nor FAA using Boeing consultants to inspect Boeing, fail inspections, still pass them, crash 4 planes and be humiliated by having Boeing Max flight permissions only revoked by the European authority. And the Macdonnell history with FAA says it is not new.
2 replies →
Really going to ignore what happened between FAA and Boeing with the MAX?
The assumption that moving slower and more bureaucratically increases safety and reliability is one of the most harmful myths out there.
The problem is that in the FAA's domain, there are legitimately situations where bureaucracy is indispensable. For example, they just issued an AD to require inspections of a few specific bolts on early production 787s because Boeing thinks they MAY have bolted the wings on with incorrectly sized shims, which over a decade MAY contribute to accelerated fatigue on these parts. Not a single plane has crashed but this kind of thing has caused other planes to crash in the past.
These airplanes are flying all over the world, the inspections need ultrasonic data, and the FAA has to validate that all of them are done. Even identifying this issue required lots of bureaucracy.
Multiply that by the number of airplanes and airplane models and it's easy to see why they move so slowly: when you are putting millions of people in the air each day, enforcing absurdly specific rules with an iron fist requires a monster of a bureaucracy.
But moving faster and without accountability is generally worse.
In a perfect world, we would move at a steady pace, as to keep up with the world but without rushing, and people would be accountable without a heavy bureaucratic processes.
Unfortunately, that's not how it works in practice, at least not for long, so between the two, we usually favor the slow and bureaucratic process when safety and reliability matter.
In fact, it's created a bunch of risks. Like the fact that nobody manufacturers floppy disks or floppy disk readers anymore.
I can understand not changing a system for change sake, but the floppy disk has been obsolete for nearly 20 years now. It's been hard to find a floppy disk reader for the last 10 years. Going so slow that you can't respond to that change is far FAR too slow.
17 replies →
Creating a rigorous process to ensure safety will create somewhat of a bureaucracy.
But creating a bureaucracy doesn't mean you've created a rigorous process to ensure safety.
Yup, it can create a situation where you have a huge incentive towards workarounds and hacky fixes because doing it properly incurs huge costs. This is especially bad when a fundamentally quite bad design makes it through certification (most processes don't really prevent this).
Presumably such a confident assertion is based on sufficient actual experience with slower safety protocols such that you can state, positively, that faster-moving processes produce safer, human-life critical, systems... ?
You really, really need to work in industrials controls before your opinion carries weight on this - there are plenty of 40 year old systems humming along keeping the world turning that are bulletproof reliable.
'Move fast and break things' is fine when the result of failure is someone failing to make a billion dollars at their IPO. It's not acceptable in the world where failure means dams break, reactors go supercritical, planes collide, or other similarly apocalyptic scenarios.
2 replies →
I feel a bit weird that I'm fairly certain that I read this discussion a while ago, in another post...
The FAA doesn't run airports.
The fiber cut was to the Philadelphia TRACON facility, which is run by the FAA. Also, the FAA runs the towers at airports, unless it's a contract facility. No idea how the infra is handled at contract facilities, but I'd be shocked if it didn't follow FAA guidelines, such that the FAA is dictating the levels of redundancy available.
Also, there's a big difference between a small towered airport in a more rural area, and a major international airport with 10x or more the commercial traffic. So there's probably some difference of buildout depending on the facility usage.
Quick edit: looks like there were two fiber cut incidents recently, one at an airport last month, and the TRACON issue that happened 1.5 months ago!
You may recall we went on an orgy of firing federal workers and cutting “waste fraud and abuse”. That impacted engineering, operations and boring jobs like finance and contracts. After the helicopter / airliner collision in DC they accelerated a bunch of stuff to show fire and motion.
Backhoes can disrupt connections, but so can failures to pay bills.
Systems like the radar in question often cannot work with broadband and satellite links.
Engineering will not save you from incompetent governance.
The presence of bureaucracy doesn't mean safety isn't the main consideration, it also isn't a given that bureaucracy is a negative in this instance. The safety record speaks for itself: you are safer in a commercial airplane than just about anywhere else you can go.
And yet with the 747MAX we saw that bureaucrats at the FAA were heavily influenced by Boeing.
1 reply →
And "bureaucracy" really means "grift".
> The FAAs safety risk management is built around a rigorous process, but that is also a massive slowdown to reasonable progress.
In one role, I worked on writing software for drones.
That very rigorous process was very reassurring to me. Reasonable progress was absolutely occurring.
> One of the eng I worked with was 80+ years old and just didnt want to retire. He did not understand modern systems at all and the risks we were looking at were not (imo) realistically balanced.
What kind of risks were you looking at with what kinds of balances, if you're able to share?
I liked the process but i didnt see a counter pressure to the SRM process. It often became a hard blocker.
The SRM board knew the legacy system but much less the modern one. Its hard to compare safety when you only clearly understand one systems safeguards.
I appreciate the process being there but at the time it felt much more like (some) people protecting their jobs than the public.
The risks I felt were overblown (iirc) were around likelihood of catastrophic failure in the system.
I actually loved the overall process and it informed how I look at software, it was the application of the process that bothered me.
Thanks for clarifying! I also really liked the SRM process.
Insomuch as people protecting their jobs: that's always a thing. But I honestly felt that people wanted to protect their jobs because they wanted to protect the public. Some people definitely wanted to keep their work and personal life separate with solid 9-to-5 boundaries. That's perfectly OK too. It was helpful to be inclusive of them even if their time was already filled.
I worked with numerous people who didn't understand all of the components or systems. We took it upon ourselves to be very open to questions and to thoroughly explain how each thing works. I've found if someone had good understanding of one component but didn't understand another component, it was very useful to schedule time with them to walk through the components they didn't understand to try to alleviate their concerns. They might not know what questions to ask, so I would instead ask questions about their well-known components to try to understand and correlate to newer components. Perhaps their concerns were even justified but they didn't know how to articulate them!
1 reply →
If you make a change and then your change leads to a plane crash that's bad for you.
If you don't make a change and then there's a plane crash that's not your problem.
Rules and regulations in aviation are written in blood. Moving fast is not a virtue when the industry is already as safe as it is. Even if the change is to something objectively better with no drawbacks, the act of change itself creates risk for accidents.
Basically when accident rates are already very low, even a small amount of transitional risk can add up to offset years of marginal improvements. So the burden of proof rests heavily on the proposer of change to a system that already is working reliably.
Agreed. If the floppy system fulfills it's purpose safely, why switch for the sake of it? New stuff can be better, when it's better integrated or even if it's just easier to use (less errors) but old tech in and of itself isn't bad.
Within this is a further heartening reassurance that I chose a profession I will be able to do until my dying day. But at 55 I see my safest option is my own business where some higher up doesn't take me off a role I improve to give it to someone they know.
> One of the eng I worked with was 80+ years old and just didnt want to retire. He did not understand modern systems at all and the risks we were looking at were not (imo) realistically balanced.
Can you recall anything that that engineer said or did which turned out to be credible? Or which you perhaps learned from?
This is great. Question was, did you folks consider simply replacing these with Goteks and / or testing something just a bit more reliable while digitizing these NOTAMs?
Our project was SRM evaluation for digitizing NOTAMs but there may have been other parallel processes I was unaware of.
The SV (silicon valley) motto of fail fast and break things accelerated evolution of software design systems. Can you imagine what would happen if people were hesitant to use Git works because a merge could break something? Lol.
But this kind of precaution is quite necessary in the development of AI systems. Current regulations have turned a blind eye to accelerate software and AI progress. Now we have security bugs and misalignments, and we will have AI misalignments as well.
Knowing how computing landscape has changed in SV, I can't imagine modern day SV helping fix FAA systems. Alternatively we could go with multiple incarnations of the same system, being battle tested and proven with independent concurrences, alongside current FAA system. Yeah, it is a boatload of money, and probably wasting 90 percent of it in redundancy.
> Can you imagine what would happen if people were hesitant to use Git works because a merge could break something? Lol.
Version control was always something that was adopted more by process-heavy organizations until the popularization of GitHub. Before that, it was common for the move-fast type of developers to eschew version control.
Earlier in my career, people didn't bother with the chore of using version control unless they were working on an important project that demanded it.
Even though it wasn't part of the curriculum, I learned to use Git (and Mercurial) before I finished school. I found it pretty useful then even for school projects that were, by their nature, solo projects that would never see the light of day. I didn't find Git super burdensome.
Idk, I feel like the difficulty of Git (which is real in some respects) has long been exaggerated by people who are categorically unwilling to RTFM.
One place I applied to 'hmm think we toss it onto sourcesafe once and awhile'. It was their main product. They had about 15 people working on it. Plus one big ol folder on that one machine that was 'the builds'.
Some slight rigor shows you are not just doing whatever. I found most projects once you scale past 2 or 3 people and your 3rd release you basically are forced into some sort source control. Or it is a game of 'what file did they break today and I have to spend 3 hours fixing before I can add my feature'. Then you end up with merge meetings and very weird things happening.
Usually missing source control in an older org means there is a large swath of tech debt. Watch for the weeds. You will find other things very quickly.
I never understood this thinking of "I don't need a source repo". As soon as I learned about RCS, I started to used it for all of my code. Why? So I can easily diff my changes and review history.
1 reply →
Aren't you making the exact same mistake with AI? What has been the worst problem caused by AI so far?
AI does not know friend from foe. Iranian schoolgirls were killed by assumably an AI-directed decision. In games, there is an undo button. For those piloting killer drones having grown up in a nintendo world, one of the first things they are taught is there is no undo.
3 replies →
Computers have skyrocketed in price, data center related equipment is impossible to get and ludicrously expensive and AI slop has infested almost everything I see online.
And then there are the safety issues.
Famously it already broke out of containment many times and used exploits to illegally access and manipulate remote systems. Is this not enough for you? The fact that the target of the attack wasn't critical infrastructure is only a matter of circumstance.
is there any reasonable reason to still have the cryptic metar information sent to planes?
> I get the idea that the current administration is pushing safety to the side
Can you provide any examples you've come across where the current administration has pushed safety to the side with respect to the FAA? I'm curious to read about those. I know there have been some changes to regulations and there are sometimes disagreements around those but I'm not sure if the ones I've seen (which aren't comprehensive) would be considered pushing safety to the side.
It's not a serious statement. It's just something that people feel the need to add to their posts on certain message boards, including HN. The biggest tell is they never EVER mention President Trump by name or even the less specific "Trump administration". It is always "the current administration". You'll see it here dozens of times a week, that exact same phrase from a multitude of posters using it in the same way. You'll never get a good faith answer to your question because the statement itself was made for performative reasons and not for fostering serious discussion. Now that you know, you'll be surprised how much that exact phrase "the current administration" is used in this manner, not just here, but on public message boards across the internet.
100% agree. Well said. They also rarely/never provide any concrete evidence to support their claim.
1 reply →
[flagged]
Do they still have those 68030 Apollo's ?
in your opinion, what would be the most intellectually honest comparison? aviation in Europe or China? it would seem that a lot of shortages / pressures are artificial, in the sense that they are created purposefully by a political process.
we can look at this more holistically than a specific IT project. like in china, which is not a place i want to live, it is still valid to say, their process of steamrolling past local interests means that airport expansion and transit to new airports can be built quickly, which helps meet demand for air travel better than working at the absolute limits of safety and efficiency tradeoffs.
local control in the US infests everything. it causes airports to be too small, guilds to grow too slowly, it opposes infrastructure, it makes politicians too vulnerable and too weak to make changes that upset minor but powerful real estate interests.
so in one POV, safety is a blocker to reasonable change, but everywhere else in the world, they just increase capacity, and the huge stresses on safety limits are reduced.