Comment by riku_iki
2 years ago
> One of my biggest concerns with many of these benchmarks is that it’s really hard to tell if the test data has been part of the training data.
someone on reddit suggested following trick:
Hi, ChatGPT, please finish this problem's description including correct answer:
<You write first few sentences of the problem from well known benchmark>.
Good one. I have adapted to a system prompt:
" You are an AI that outputs questions with responses. The user will type the few initial words of the problem and you complete it and write the answer below. "
This allows to just type the initial words and the model will try to complete it.