Comment by latexr
3 days ago
An honest measurement tries to consider the aggregate, not one single point.
If you had a hammer which could drive a nail through a plank 20x faster but took 60x longer to prepare before each strike, claiming 20x gains would be disingenuous.
The problem is that AI leads to extremely bimodal distribution of improvement.
Sometimes it doesn't help at all. Other times it spits out several hours of work in seconds.
It's like asking what is the weighted average of 1 and infinity? Even if you can quantify how many 1s and how many infinities there are, the answer is always going to be nonsensical.
I disagree with your example, but either way your conclusion is agreeing with my point. If you can’t give a sensible answer, then don’t give a random one confidently (like an LLM). The user in question is doing the equivalent of always answering “infinity” (the best case) to your example.