Comment by mlmonkey
15 days ago
Sadly, this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology, "don't be evil", etc. And Google is run by managers now.
Expect Google models to be benchmaxxed as much as possible, but don't expect commensurate performance on real-world tasks.
>> this is all that managers know to do: take a metric and try to optimize the heck out of it. They don't care about research, technology
This is not true. It is needless negative hyperbole that IMHO doesn't belong on HN.
I'm speaking from 30 years experience in the Valley, having worked at several big companies.
While this might be overly cynical, it's also a powerful validation of the exact feedback loop that you want to have in an organization.
If what you say is true, then the next set of benchmarks will be written to illuminate the true reflection of real-world tasks. Then those get benchmaxxed so managers can show "numbers go up."
Repeat enough times, and you get materially and genuinely improved products.
The burden is on the press, outside researchers, and general tech industry to both write and care about the metrics that matter.
There's just one problem - culture eats strategy for breakfast. You cannot govern culture or improve culture with metrics.
In large companies (Fortune 1000 etc.), you don't have much of a choice, the culture was eroded long ago, and corporate governance by metrics is a lesser of evils. But thankfully, not all companies are there yet.