Comment by kaffekaka
5 hours ago
It is a very human trait to believe/feel "what I do is special so general rules do not apply". The bias is very strong.
The same way people have opinions on things like for example wines without actually being able to discern them in a blind test.
We humans want to think we can have strong, well founded opinions even when we cannot.
I'm confused about what point you're trying to make. Are you saying that there is no discernable difference in challenge between different software projects – say, between a stereotypical todo app and a physics simulation engine? Or are you saying that most people falsely believe they're working on the latter while really doing the former? Or something else?
My read is that GP is eluding to people over estimating how much of a model they need for a task. Perhaps Claude can make more features work in autonomous mode more often, but what are the actual artifacts? What is the quality of the architecture?
Their point may also be (re: wine), that some devs have been conditioned to think a certain token vendor (vineyard) is of superb quality, while if given a blind test (tasting) of the agents on a real world task, they would not notice the difference.
For those of us who use open weights, the difference has narrowed dramatically this year. I've personally been mostly eyes-free developing an NES harness with mostly flash models, bringing in Kimi from time to time, and stepped in a couple times to reframe the approach or architecture. For those of us who (when we) stay in the loop, the difference is near-zero because the tasks we give them are more bite sized.
The difference I have observed is most closely tied to the complexity or context requirements. While I work in a multi-million LoC codebase without any issue for open weights or flash models, my NES project is only 20k lines. The challenging part is the core loop (which needs to blend the emulator FPS with the agent loop) and game state observation (needs to detect byte changes in emulator RAM). The ROM decoding itself has not been so bad, I have actually been quite impressed with the little open weight models on this front. After it extracted the maps, then pulled out even more details, it has written an A* implementation that requires moving between different modes of transportation. (Final Fantasy 1)