Comment by aspenmartin
5 days ago
Your observations are right but pretty insane to consider them a pure PR company lol. They are making more frequent releases so yes the release-to-release quality is smaller but we’re still ascending quality and reliability curves the same way we have since GPT-3. You get a GPT4->5 leap every like 17 or 18 months I think it is
The gradient of improvement is absolutely not the same.
If anything its slightly higher. Feel free to provide any evidence to the contrary.
ECI (good aggregate measure using IRT): https://epoch.ai/eci?view=graph&tab=release-date&subset-view...
METR time horizon (now topped out): https://metr.org/time-horizons/
I like this one, although its data seem to overlap with ECI.
https://artificialanalysis.ai/trends