Comment by codingjoe

Maybe we need more benchmarks that involve human collaboration instead of solo task solving.

Since they have become so capable the new bottleneck is what they can't know. The stuff that's inside people's brains who work in real companies with products and processes absent from any training set.