Comment by simonh
5 years ago
You could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail.
It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.
For some reason people think everything is as good as it can be at FAANG and other big name tech companies, and everyone else just walks around with their pants down at their ankles bumping into walls until 5:00. It’s just not true.
I mean, it's not just FAANG. Literally everyone except banks use the standard Python environment. simonh is right about the reason for the forked tech stack.
Actually, it's probably not FAANG folks at all. I'd expect ex-FAANG folks to be more sympathetic to the forked python situation... FAANGs have an abundance of non-standard and frustrating infra (wasn't 5TB just posted yesterday?), and maybe even on steroids compared to banks (do any of the FAANGs not have at least one custom linux kernel?) Hell, both As roll a shitload of their own silicon.
Exactly.
Facebook literally maintains a python fork: https://github.com/facebookincubator/cinder
Google invented "NoSQL" before anyone else knew what it was, and all those "cloud" tools they used internally were obviously proprietary (except the ones they open sourced). Ex-Googlers I work with typically had to spend quite a bit of time re-adjusting to the "inferior" tools and processes in other companies.
Microsoft invented their own development ecosystem, and the only reason it's "common" or "standard" in the tech community is because they sell it as a product. This is the same for Apple at least for iOS development, and Amazon for their cloud service offerings.
When companies have millions of dollars to spend on maintaining a custom development environment that they think will give them a competitive edge, they will do it. It's the smaller shops that can't afford not to go with the flow, so to speak.
I mean Google has Bazel which has reliable hermetic distributed builds with modern statically typed languages. I don't think this weird custom Python system is really in the same league. Barely even playing the same game.
Well, I don’t know about Quartz. I worked with DB systems and they were awful in that regard. They worked, but largely because people stuck to convention.
For example, changing scheduler jobs required submitting a change in Excel and having it approved (twice...) by someone. Except the table was world-writable and changes not logged. So in principle only your appropriate superior could approve change, in practice anyone could, and you’d never even know.
Youch, that's nasty. I can completely believe it though, banks are huge unwieldy organisations. I spent a fair bit of time working with auditors though, so I know a huge amount of effort goes into rooting out things like that.
The thing is just because a team in a bank did this thing, that doesn't mean "The Bank" thinks that's a good idea. Like any company, banks are communities. I'm not making excuses, the fact this system wasn't properly architected is a failure of governance, but I've been on the other side of this trying to get teams to fix their problems and adopt resilient processes and procedures. Every offender thinks their service is special and their violation of the standards is justified.
> Quartz has ... solid tooling
Is that how you'd describe the IDE and (integrated) VCS?
Yes, it's fine IMHO. Ok it's not your favourite Java IDE, but it's way, way better than some of the crap I've had to use at various places. But then I wrote a 5Kloc PyQt desktop app almost entirely in IDLE so yeah, maybe I'm not the best judge.