Comment by afavour
3 hours ago
How would you be able to know that N+1 hasn’t scraped a source containing the questions asked of N1 and optimized its answers ahead of time? In many ways that would be the logical course of action when you’re tasked with creating an LLM that is assessed by criteria X/Y/Z.
This is not an intelligence problem. It is in part an operational problem (have you properly set up the test), but it is mostly an alignment problem. Is the model cheating or not. Alignment is, indeed, very difficult.