Comment by evilmathkid
5 days ago
I didn't phrase that well and cant edit, so clarifying:
What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
>What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the exam"
This is getting into metaphorgotten territory, I'd have to do a deep dive to really understand whether it is "cheating" or, more formally, a methodology that encourages overfitting.
But for what is worth, taking the textbook to the exam (regardless of whether you were born there or not), would be cheating in a test as well. Although it is possible that in the model training context it does not lead to overfitting, it certainly doesn't preclude it.
>Yes what you describe would be cheating. My approach is the opposite. What I did was "Carry your textbook to the exam and then learn from scratch during the exam"
It is comforting that someone agrees, to this day this is considered a gray-area (or not even that) by students, graduates and those that look up to the University of Buenos Aires institution.
well not if its an open textbook exam
in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc
this is one such case
--
the separate overfitting concern is fair, look at private holdout performance for that. it performs on par with TRM (a comparable model) in the private set, ofc with far less compute
I think you can do whatever you want with your train set, what would be concerning is training with the test set.
I don't think overfitting is separate, the consequence of putting the benchmarks in your training set is not that you "cheat" by breaking some moral code, it's that it breaks the purpose of the benchmark and trains your model to be good at that benchmark only, instead of being generally useful.