Comment by menaerus
2 hours ago
I'm not sure what you mean by "running a job that ran a worker". Running the system is vastly different than building and maintaining one. I work on highly available and distributed systems spanning multiple data centers, and single node scalability of database kernels, so I guess I could also say that I've done and still doing work at the highest level but in my case it is the core development. Which one is yours?
The system I built was called exacycle. It ran as a borg job in every cluster we had access to, with priority 0 or 1, with N tasks (where N=# of machines in the cluster). Borg ran the "worker" task as long as there were resources free on the machine (often, several workers ran on the same machine, and no workers ran on other machines). There was a global schedulers that picked mini-tasks off a global queue and fed them to the cluster-level scheduler (batches based on instant resource availability, and subject to some data locality optimization) which then fed them to the machine workers.
At peak, we had 1-3 million xeon cores running our tasks (protein folding, protein design, drug docking), 24/7. This was before GPUs were really a thing. Each mini-task was basically a rosetta or gromacs or autodock binary running on local inputs and writing local outputs for ~1 hour using ~1GB RAM and no network. Input and output was via colossus, or distributed through a global torrent network (literally all 1-3 million cores formed a single data transfer group, with data passed between clusters at low probability).
I built most of the code of the system (which depended on Google infra) including the scheduler and the machine worker, and also ran the system as on-call "SRE" for 3 years. During that time, we nearly almost didn't take down the Google global network, and had small latency impacts on many services. Managing to run the system for so long without antagonizing ads or search or youtube was a major challenge.
I would say the complexity of the system I built (considering everything down to the OS) was roughly the level of a few human cells in a plate, in terms of interacting components and the complexity of their interactions.
Here are some of the publications that came from running exacycle (we provided strong evidence that using more compute improves outcomes in molecular simulations)
https://pubmed.ncbi.nlm.nih.gov/24345941/ https://pubmed.ncbi.nlm.nih.gov/24265211/
I appreciate the details, very interesting. So from a quick glance my understanding is that exacycle is a task partitioning and dispatching system built on top of existing google infra runtime (borg). Tasks are units that are massively parallel and distributed across the compute, and have no or almost no shared data. Problem space seems to be built around self contained tasks which have well defined input and output, and main engineering challenge in the exacycle itself would be the optimizing the system for maximum resource utilization? It's complex but not as complex as open problem space engineering problems dealing with highly concurrent and distributed workloads with shared state and that are unknown at the time you're designing the algorithm.
Anyway, my point was not in that I would equate software complexity with the complexity found in biology but that I see the same trend and similar type of challenges that can be found in both fields and therefore I see both fields amenable to the AI advancements whereas many people from the thread do not, and I disagree with that.