Comment by stillpointlab
2 days ago
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
That is similar to saying to Magnus Carlson that he should just increase his understanding of chess.
The point of my comment is, no amount of "increased understanding" puts a human on equal footing to a chess engine.
> non-intuitive alien code being generated by agents
You're saying it as if humans can't write that too.
I am saying that we won't be able to write it, in the same way that Magnus Carlson (or whoever your top chess player would be) cannot compete with a chess engine.
I am arguing that, much like the "best" chess move is unintelligible to best chess players, truly secure code will be unintelligible to humans.
To make good code intelligible is compiler's job