Comment by dartos
1 month ago
I mean all optimization algorithms do is game a benchmark. That’s the whole point.
The hard part is making the benchmark meaningful in the first place.
1 month ago
I mean all optimization algorithms do is game a benchmark. That’s the whole point.
The hard part is making the benchmark meaningful in the first place.
Yeah, and if anything, RL has a rep of being too good at this job, because of all the cases where it gamed a benchmark by picking up on some environmental factor the supervisors hadn't thought of (numerical instabilities, rounding, bugs, etc.).
My favourite is this one:
https://news.ycombinator.com/item?id=43113941
The ML version of Professor Farnsworth[1]:
It came to me in a dream, and I forgot it in another dream.
[1]: https://www.imdb.com/title/tt0584424/quotes/?item=qt0439248
No, that is patently false. Many optimization algorithms which computer scientists, mathematicians or software developers devise do not involve benchmakrs at all, and apply to all possible inputs/instances of their respective computational problems.
Plot twist: the loss function for training is basically a benchmark
Is it a plot twist if it’s the whole plot?
Example?
Those times when people write code with loads of theoretical micro optimisations that they never actually test against a benchmark because otherwise they wouldn't do it.
2 replies →