Comment by rtpg

I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".

And maybe it does a single finalized clinical trial.

You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).

I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.