Comment by ben_w
15 hours ago
Benchmaxxing is the default case, and always has been.
It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a problem in machine learning, it's the standard failure mode of all minds capable of learning, human, animal, artificial.
Even pure genetics has this problem. Viruses and cancers also demonstrate this behaviour, with the bench being evolution's only option: reproductive success.
Why would I adapt myself to unseen environments unnecessarily?
You can see possible existing environments even if you explicitly blind yourself to them, through reflections off of environments that you do not blind yourself to. Even with the benchmark excluded, the social zeitgeist that has considered the benchmark and included it or ideas from the benchmark either implicitly or explicitly in their code, documentation, et cetera is still part of your training data.
Because then you could sit under the palm tree and enjoy your free bananas and relax.
Propagation
You can't justify it being OK just because it's common. Here, this just makes benchmarks into a low signal and useless marketing number once people get numb to all the 99%s.
Also no idea what viruses and cancers have to do with this. Cancer is surely very poor reproductively because they never spread to other hosts.
Cancer is simply cells breaking free of the cooperative jail. Essentially the grey goo scenario of nano machines. Instead of cooperation they just do their own thing.
Cells have a certain optimised DNA mutation rate kept in check by various machinery. Multi cellular life expects each of these little replication machine to co-operate in the grand scheme of running a body. But it's also required in the grand scheme for DNA to mutate a little bit to ensure population variance. So you could say that cancer is the tax paid for having cooperative yet flexible and adaptive nano machinery.
So yes the propensity for cancer developed under evolutioniary pressure towards a non zero level.
The population could have optimised for zero cancer but it would not have paid for itself in terms of overall population adaptability and survival.