Comment by beachy
5 years ago
> Due to the core software limitation (no source code available) we were compelled to integrate some original PDP-11 computer cards into the final product and this resulted in a hybrid PC platform.
Amazing. I've never been anywhere that lost their source code. Would love to know how it happens.
I have been in two different situations many years ago, neither of which I take responsibility for, ha-ha:
1. Contractor relationship gone sour, turned out they had withheld source code from us. This lead to a dispute regarding the code ownership. Meanwhile we were running the corresponding binaries in the wild. We ended up reverse engineering the missing parts.
2. Source code is centrally managed on a server, hard disk failure, restore not working or backups wholly missing, don't remember. This was the pre-git era, probably Visual SourceSafe, which was entirely centralized.
For me personally, let's just say I feasted my eyes and learned a lot. But I wager that there is still a lot of companies vulnerable in very similar ways today.
Sourcesafe is a 90-era product. PDP-11 was discontinued long before Sourcesafe was even released.
There were version control software around in the 70’s but I do not think that the adoption rate was very high. I think it’s reasonable to assume that they didn’t use it in this case.
The SCCS paper (https://ieeexplore.ieee.org/document/6312866 ) says the adoption rate at Bell Labs was high.
> The Programmer's Workbench has proven to be very popular with both management and programmers, and is now used by almost all software projects at the author's installation.
The followup paper https://dl.acm.org/doi/10.1145/800283.811111 at https://dl.acm.org/doi/pdf/10.1145/800283.811111 comments:
> SCCS is used extensively within Bell Laboratories. For example, at one installation more than 3 million lines of source code are controlled with sccs. These 3 million lines represent 5000 files and the work of approximately 500 programmers. Additionally, sccs is used at many installations outside the Bell System.
6 replies →
A C++ project I inherited had some binaries checked in for rarely-updated utilities - no sense in forcing everyone to build everything from scratch every time, right? Unfortunately, this wasn't done by CI, but by hand - so while those binaries could be built on the original maintainer's machine, some stale & missing source files (which said maintainer forgot to `p4 add`, `p4 reconcile`, or perhaps submit?) meant the only buildable copy of source was on their workstation, without anybody noticing.
...which was then presumably lost/reformatted some time after said dev left (quit? laid off? fired?)
Unable to build said source code, a subsequent developer added a python script to postprocess the output of said binary for their own needs. A hack, but it worked.
Then I got my hands on it. Well, I needed data that was discarded by the binary, so hacking on the python script was a non-starter. Instead, I reverse engineered what the binary was doing by stubbing out enough of the source code to get it building - then compared hex dumps of the outputs and tweaked said source code until my modified source code gave the same output as the checked-in binaries. Then I got rid of the python script by moving the logic into the binary's source code, where it really belonged. Then I made my own changes.
Didn't bother setting up CI though. Was porting a project in a heavily compressed (<1 month!) timeframe, and nobody else needed to touch said binary, and the codebase wasn't going to be used for any future projects (it was pretty terrible, and much time had been spent creating a much better codebase for future projects.) I at least made sure I checked everything in, though!
>> the codebase wasn't going to be used for any future projects
I have heard this many times, especially for the "single use executable for a specific use case". Many years later, it had made its way into production code and is on its 10th iteration.
The software package they were using was an Ericsson product, called "JZA 715", for PDP-11s. Did Ericsson ever give them the source code? Possibly not.
Does Ericsson still have the source code for a PDP-11 software package they sold in the 1970s? Maybe they do, maybe they don't.
Did they ever even ask Ericsson for the source? Possibly, they may have thought that modifying PDP-11 software was too hard–people with skills in doing it are hard to find these days, it is unlikely to be written in C, probably either assembly or some obscure language few have heard of–so even if Ericsson still had the source and was happy to hand it over (free of charge or for a reasonable fee), they might not have thought it worth their while to acquire it, and hence never have asked Ericsson for it.
EDIT: Actually, turns out JZA715 is mostly written in Pascal (so not that obscure a language after all) with some modules in PDP-11 assembly – see page 18 of http://srsv.org.au/wp-content/uploads/2017/02/S-38-1-Jan.pdf – the software ran under RSX-11M
Surely they could use a disassembler and from that (and monitoring the i/o of a live system) infer a test suite from which they could rebuild the API. The new system could be tested to have the same outputs of the live system, and eventually swapped in, fixing bugs and making it more reliable in the process. But I guess that was more expensive than the FPGA emulator
Reverse-engineering of even simple binaries tends to be difficult enough, without the stakes being as high as the program safely running the train control system of a major city. I can't pretend to know what conditions this was originally engineered under, but these kind of systems today are subject to intensely rigorous testing, and sometimes formal analysis. In a safety-critical system it's just not going to be worth doing that. Plus, if the original source code has been misplaced, the original requirement documents are probably gone too.
I sort-of kind-of agree with this. It is certainly why most of these systems aren't worth upgrading. But reasons why it might be desirable to wear this cost is if the original system is already showing signs of bugs, which may be true if it's being operated in new conditions (e.g., interop with new systems, increasing workload), or there are safety features that could be added but not without refactoring. But, yes, if it ain't broke, surely there's no need to fix it
Edit: Also worth pointing out that formal verification was likely not a component of building the original software, and it likely has bugs it's kept since inception. Updating the API to a language more conducive to formal verification might be an option that improves safety over the legacy system
Edit: Apparently they did replace it all eventually
https://news.ycombinator.com/item?id=29158255
When you have government representatives that don't understand technology, it doesn't get funded.
It's no wonder everything is emulated/old/never-replaced.
In defense of government, they need to have an audit trail and (in functioning democracies) can be held accountable when things go seriously wrong, so it means they only have a select few orgs to choose from that have passed all the certifications/due diligence. Connex (the company running the show at the time) were a private operator who has lease of the rail system for probably long enough to justify modernizing some of the systems, but maybe not enough to justify a full rewrite. Rewrites of systems with big safety implications, like civil transport, would have a different risk profile to upgrades of less consequential legacy systems
Edit: Apparently it did eventually get replaced https://news.ycombinator.com/item?id=29158255
Breakdown of communication among key stakeholders, and nobody sufficiently motivated enough to stick their neck out :)