Comment by piker

Go read Karpathy on this, his telling will be more nuanced than mine for the specific example. But the whole point of NanoGPT is that it’s distilling all of his wisdom and taste into a very dense and maximally educational version that fits on a single screen. It’s simply not something that has been done before in the corpus.

Planet scale infra often requires some novel ideas at the architectural layer (the engineers can input those) and then it’s mostly in-distribution C++ / Rust / Go; most of the hyperscalers open-source their stacks, for one. But for two, a ring buffer, look-aside cache, deterministic hash, b-tree, lsm-tree, etc. are all well-known patterns.

Hyperscaling is also often a very clear objective function; i need this code path to run in this many microseconds/nanoseconds, so i can hit the scale numbers I need. Claude/Sol can extract prod logs and build a representative micro-benchmark, and then hill-climb on it autonomously. Thats straight up the fairway for the training set, even if it often has a high bar for finishing and requires sophisticated Workflows or lots of tokens to explore the search space.

E2A: sorry sorry, typo, I meant microGPT. I can see why this would be confusing.

https://karpathy.github.io/2026/02/12/microgpt/

A lot of stuff that underpins planet-scale infra is open source and has been for a long time. The general approaches and architectures for it have also been widely discussed and litigated in the commons so it's not just the code - the specific why is also very well documented.

So despite its importance much of it is actually pretty in-distribution.