Comment by nullbio

Wild that we still haven't figured out how to make good benchmarks. What we really need is a way to properly quantify what makes a codebases architecture good, and then evaluate architecture of generated codebases, or evaluate refactors of existing ones.

Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.

None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.