Comment by curt15
4 hours ago
> How much can we trust a system when nobody has experience of building it this way?
If humans are no longer reading the LLM outputs and internalizing the codebase themselves, there's no longer much point to source code. The coding assistant might as well take a natural language prompt and directly hand you an ELF binary that passes a test suite. Are companies ready for that?
There is a point to source code:
Lots of tools for analysing and debugging it
It can be cross platform - the same source code will run on x86 and ARM, and in many cases on Windows and Linux.
Changes have predictable consequences because compilers and interpreters are deterministic.
I think some definitely are. Especially if there is a system that sits on top of that and pokes and prods the binary to see if it behaves like it's supposed to.
Gary Tan has said markdown is code. Personally I see my job as establishing the architecture, creating tests, lint rules, etc. for my agents. Monitor the code, but if there's an issue create a new agent that monitors performance, etc.
That's simply not true.
Until spec files can be worked with so efficiently that we can just regenerate the binaries from them for every change, what you said makes little sense.
You're going to need an intermediate portable form of storing the program. It so happens that all the tools and AI are optimized to work with source code.
Exactly, at what point do spec files become so precise that they are just another form of source code?
Probably at a point where my worries will be when my team of construction robots finishes my villa, rather than whether companies are prepared to ditch source code.
Exactly! I wouldn’t be surprised if some day in the distant future, that’s where this all goes. Coding languages may in fact be on their way to being dead. You can imagine a world where, regardless the platform, the lowest level instruction format is output.
I can see a future generation of models. For now, LLM's train on human artifacts and they handle Python/TS well because of the overwhelming amount of code they've seen, combined with the semantic labels high level languages use that they rely on for reasoning.
Also, consider token efficiency. Imagine how many tokens would be torched just translating an import statement of a popular library to its binary equivalent, let alone a big complex program
How would we train those LLMs? Feed them binary executables along with their business requirement docs? Is the corpus of training data large enough to be effective? What happens when a new CPU architecture is developed? Where would the training data come from for the new machine code?
Exactly! That's why the "pelican riding a bicycle SVG" test is unfair. It's a token predictor, not an intelligence. What corpus of SVG data has the LLM been trained on? Only in the last 10 years has "drawing using data" been in existence. Midjourney can generate you a pelican riding a bicycle artwork easily. But instead, if the test is intentionally based on a task with no corpus, then you get garbage.
I would not be surprised if we ended up with binaries and Lean proofs.
isnt that kinda like downloading and using software a team you have no knowledge of has produced and released?
All you can go by is 'does it do what it says it does'
Yeah, all these devs pretending you can only trust code you understand, then going and pulling in 1000 random npm deps...
When people install npm packages, they typically only install ones the community trusts. These packages have clear APIs, open source, good docs, test coverage, security checks, and, importantly, usually no undocumented side effects.
Engineers introduce these packages into their own projects that have far fewer of the above guarantees than OSS, are built under pressure from deadlines, and have a constantly churning set of maintainers.
I think from the perspective of a business its own code does call for greater scrutiny than its dependencies.
1 reply →
I feel like ever since we stopped reading assembly we don't really internalize our codebases.
It's just one "Buy Now" button. How hard could that be? /s