Comment by skybrian

I find collecting “gotcha” questions helpful because when a new model can answer them, it shows improvement, and if it doesn’t, it’s a reality check that, despite the model being helpful for many tasks, there are other things it can’t do yet. It’s a demonstration of “jagged intelligence.”

It doesn’t have to be a negative thing! Simon’s pelican on a bicycle prompt is an example of a “gotcha” question.