Comment by btdmaster

There is a cool solution for this: https://huggingface.co/spaces/Jellyfish042/UncheatableEval

This doesn't work for instruction-tuned models, but it's an interesting alternative approach that doesn't need a complicated (and thus gameable) evaluation function or human interaction. Instead, predict the next word with data newer than the training set.