Comment by alecco
5 years ago
I worked at one of the largest of these systems. It seems to be the one referred by the post.
The global distributed store of pickled python objects using Event Sourcing was one of the most horrible and expensive database systems I've ever heard of. It runs on THOUSANDS of expensive servers with all data stored in-memory. To get the state of a single deal you had to open, decompress, deserialize, and merge hundreds if not thousands of instances. And 90% of the output was more often than not discarded.
The Python interpreter extensions reveal the ignorance of Python by the original developers. There was no good reason to fork CPython.
There were many small subsystems created and supported by lone rangers with impressive CVs and astronomical salaries. A JIT better than any other one out there (but with a lot of limitations). A meta-query system extremely elegant.
But this all was a sham. The actual daily crunch/analytics was run on more classic SQL/Columnar clusters. From the distributed object database hours-long running batch jobs loaded stuff on old school DBs. And those blew up frequently. Sometimes those blow ups cost many millions in delayed regulatory reports. The queries running on top of SQL were beyond stupid and the DB engine could not optimize for them. And of course, people blamed SQL and not the ridiculous architecture and the OOP dogma.
Don't work for old school banking, hedge funds, or anything like that. They are driven by tech cavemen and their primadonnas. Exceptions might be some HFT and fintech shops.
It looks like sound ideas, poorly implemented. A lot of these capabilities crop up in BEAM or Smalltalk. I do wonder if Erlang had better developer ergonomics we wouldn't have seen that instead of Python as a basis.
But it was good for the first generation who worked there. Big dollars, total control and even the opportunity to create a proprietary IDE...
That proprietary IDE was a piece of crap. Monkey patching was prevalent, exponentially increasing startup time depending on the last time you opened it, and an “online” source tree where one could easily modify the source code in someone else’s ‘private’ workspace.
PyCharm was a move in the right direction, but the way it worked was absurd- it would run the internal IDE in the background and sync to the file system. Given the proprietary IDE took up more resources than PyCharm, you constantly had to shut down apps so the machine had enough memory.
IDEs should not be a requirement- they are tools… but you had no choice but to use it and their totally flawed code completion. Their measure of success was tantamount to having a Jupyter notebook- write code and get back results immediately.
I guess it was originally created as a productivity tool as there was not any good Python IDE back before the year 2010. But after a while it became a monstrosity and new tools emerged but it was rooted so deeply that it was impossible to implement the new tools.
But as said I'd really love to work on those projects. Most of the people don't get the prestige to own one's own baby in a big corporation. 99% of the job is maintaining a shit mountain of code and piling new shits on top of it. It really took a lot of luck to be able to make it happen.
I mean even for people who get to work for Jetbrain or Microsoft Visual Studio team, they don't get to create new IDEs, they are buried deeply in a shit mountain of code and JIRA issues.
Plus the pay and vacation is really good for the banks.