Comment by godelski

I understand how you interpreted my comment as this. That's my bad.

But even when taking state of the art knowledge as a ground truth aligning to that is incredibly hard. Medicine is a great example. You're trying to create a causal graph in a highly noisy environment. You ask 10 doctors and you'll get 12 diagnoses. The problem is subtle things become incredibly important. Which is exactly what makes measurements so fucking hard. There is no state of the art in a well defined sense.

The point is that in most domains this is how things are. Even in programming.

Getting the right answer isn't enough