Comment by sigmoid10
1 year ago
Apples and oranges. VW actually cheated on regulatory testing to bypass legal requirements. So to be comparable, the government would first need to pass laws where e.g. only compilers that pass a certain benchmark are allowed to be used for purchasable products and then the developers would need to manipulate behaviour during those benchmarks.
There's a sliding scale of badness here. The emissions cheating (it wasn't just VW, incidentally; they were just the first uncovered. Fiat-Chrysler, Mercedes, GM and BMW were also caught doing it, with suspicions about others) was straight-up fraud.
It used to be common for graphics drivers to outright cheat on benchmarks (the actual image produced would not be the same as it would have been if a benchmark had not been detected); this was arguably, fraud.
It used to be common for mobile phone manufacturers to allow the SoC to operate in a thermal mode that was never available to real users when it detected a benchmark was being used. This is still, IMO, kinda fraud-y.
Optimisation for common benchmark cases where the thing still actually _works_, and where the optimisation is available to normal users where applicable, is less egregious, though, still, IMO, Not Great.
The only difference is the legality. From an integrity point of view it's basically the same
I think breaking a law is more unethical than not breaking a law.
Also, legality isn't the only difference in the VW case. With VW, they had a "good emissions" mode. They enabled the good emissions mode during the test, but disabled it during regular driving. It would have worked during regular driving, but they disabled it during regular driving. With compilers, there's no "good performance" mode that would work during regular usage that they're disabling during regular usage.
> I think breaking a law is more unethical than not breaking a law.
It sounds like a mismatch of definition, but I doubt you're ambivalent about a behavior right until the moment it becomes illegal, after which you think it unethical. Law is the codification and enforcement of a social contract, not the creation of it.
16 replies →
unless following an unethical law would in itself be unethical, then breaking the unethical law would be the only ethical choice. In this case cheating emissions, which I see as unethical, but also advantageous for the consumer, should have been done openly if VW saw following the law as unethical. Ethics and morality are subjective to understanding, and law only a crude approximation of divinity. Though I would argue that each person on the earth through a shared common experience has a rough and general idea of right from wrong...though I'm not always certain they pay attention to it.
Overfitting on test data absolutely does mean that the model would perform better in benchmarks than it would in real life use cases.
1 reply →
ethics should inform law, not the reverse
1 reply →
I disagree- presumably if an algorithm or hardware is optimized for a certain class of problem it really is good at it and always will be- which is still useful if you are actually using it for that. It’s just “studying for the test”- something I would expect to happen even if it is a bit misleading.
VW cheated such that the low emissions were only active during the test- it’s not that it was optimized for low emissions under the conditions they test for, but that you could not get those low emissions under any conditions in the real world. That's "cheating on the test" not "studying for the test."
> The only difference is the legality. From an integrity point of view it's basically the same
I think cheating about harming the environment is another important difference.
How so? VW intentionally changed the operation of the vehicle so that its emissions met the test requirements during the test and then went back to typical operation conditions afterwards.
VW was breaking the law in a way that harmed society but arguably helped the individual driver of the VW car, who gets better performance yet still passes the emissions test.
It might sound funny in retrospect, but some of us actually bought VW cars on the assumption that, if biodiesel-powered, it would be more green.
And afaik the emissions were still miles ahead of a car from 20 years prior, just not quite as extremely stringent as requested.
2 replies →
Right - in either case it's lying, which is crossing a moral line (which is far more important to avoid than a legal line).
That is not true. Even ChatGPT understands how they are different, I won’t paste the whole response but here are the differences it highlights:
Key differences:
1. Intent and harm: • VW’s actions directly violated laws and had environmental and health consequences. Optimizing LLMs for chess benchmarks, while arguably misleading, doesn’t have immediate real-world harms. 2. Scope: Chess-specific optimization is generally a transparent choice within AI research. It’s not a hidden “defeat device” but rather an explicit design goal. 3. Broader impact: LLMs fine-tuned for benchmarks often still retain general-purpose capabilities. They aren’t necessarily “broken” outside chess, whereas VW cars fundamentally failed to meet emissions standards.
Tesla cheats by using electric motors and deferring emissions standards to somebody else :D Wait, I really think that's a good thing, but once Hulk Hogan is confirmed administrator of the EPA, he might actually use this argument against Teslas and other electric vehicles.