Comment by raducu

> out-remembering people around us.

Doing an angular -> react refactor with Opus 5 at work.

Amazed at how good it is but also amazed at how dumb it is after I told it I want a very basic and specific code review:do the file names match the contents? is one functionality/concern split in multiple files? do the variable names match what they do? does a method read nicely?

Anyway, I was very surprised at how bad it can be at this.

I suspect it's because the LLM simply does not focus well enough on a single method.

My gut modelling is that the critical files are quite small, some 4k LOC and the LLM tries to read them all in one go and my requirement is surprisingly difficult but the LLM does not realize it, loads all the input into context and simply cannot find a good signal per token/group of tokens.

I guess I can do it because I literally cannot fit 4k LOC in my head and I actually read only tens of lines but I can reeealy dig deep in those lines.

The LLM can remember lots, it has 1M context window, was trained with needle in the haystack problems, but it can severely underestimate how much computation is required with TASTE/universal computation type tasks.

Yes, LLMs are incredible when it comes to delivering well defined outcomes, it's evident they were RL trained very well. But they are incredibly bad at understanding nuance in a text.

It's incredible how agentic coding benchmaxing seems to correlate with writing good poetry but at the same time finding good names for files and variables and wiring beautiful code does not.

Same with getting basic details of who said what in an "adversarial" conversation between 2 humans -- LLMs are incredibly bad at it, with all their "almost discovering new maths" skills