Comment by wg0
8 days ago
This checks out logical speaking.
The FANG code basis are very large and date back years might not necessarily be using open source frameworks rather in house libraries and frameworks none of which are certainly available to Anthropic or OpenAI hence these models have zero visibility into them.
Therefore combined with the fact that these are not reasoning or thinking machines rather probabilistic (image/text) generators, they can't generate what they haven't seen.
No it doesn't check out. I think it's becoming abundantly clear LLMs learn in real time as they speak to you. There's a lot of denial and people claiming they don't learn that their knowledge is fixed on the training data and this is not even remotely true at all.
LLMs learn dynamically through their context window and this learning is at a rate much faster than humans and often with capabilities greater than humans and often much worse.
For a code base as complex and as closed source as google the problems an LLM faces is largely the same as a human. How much can he fit into the context window?
You're observing this "paradox", because what you call learning here is not learning in the ML sense; it's deriving better conclusions from more data. It's true for many ML methods, but it doesn't mean any actual learning happens.
There's another phenomenon. It's called pedantic denialism. Deriving conclusions from more data is the same thing as learning. You learned something from the new data hence the new conclusion. As long as that context window survives the LLM has learned.
It checks out if you take into account most developers are actually rather mediocre outside of places where they spend an insane amount of time and money to get good devs (including but not limited to FANG)
That's why coding agents usually scans various files to figure out how to work in a particular codebase. I work with very large and old project, and Codex most of time manages to work with our frameworks.
Huh? I have over a hundred services/repos checked out locally, ranging from 10+ years old to new. I have no problem leveraging AI to work in this large distributed codebase.
Even internal stuff is usable by the model because it’s a pattern matching machine and there should be documentation available, or it can just study the code like a human.
Yeah that's still very far away from FAANG repos
In total LOC sure. This isn’t close to my companies total repos either… But surely a FAANG dev isn’t writing code across thousands of repos. In fact the people I know most at fang have less scope than this not more. So what is the relevant blocker here?