Comment by jsenn
10 hours ago
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
I think you're underestimating how ruthless the AIs are at maximizing.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
There are probably more mis-aligned humans than mis-algined AI's today.
> You are picking on 1 bullet point out of 7.
We are observing the Gell-Mann amnesia effect in practice.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?