Comment by rtpg
2 hours ago
I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".
And maybe it does a single finalized clinical trial.
You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
this is not what they're claiming.
the reason why clinical trial exists, is to prevent reward hacking. either by organizations or LLMs. the reasoning about why a drug works is usually post facto discovery (if that happens at all). the R&D environment around drug discovery is hilariously lacking (my wife works in this space. sometimes my mind just blows hearing some stories.)
if the model comes with a ridiculously shaped drug that no human could understand. you could basically put through the clinical trial process (mouse -> apes -> RCT with humans.) and decide whether we want to continue or not.