Comment by dekhn
9 hours ago
I work in drug discovery and AI and his slide about cancer medicine isn't very well informed. We have models and narratives about how drugs work, but they are woefully incomplete.
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
I also work in drug discovery, and I completely agree; the extent to which the actual causal mechanisms of the efficacy of many drugs is woefully short of some human-appreciable first-principles based explanation. The point is especially undercut by hypothetically suggesting that this drug /does/ in fact pass stage 3 trials, at which point I can't fathom anyone would have a problem advancing it.
Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
The accounts spreading that narrative on X no doubt have their own agenda. Before people start hyperventilating about "AI" conquering human endeavors, remember that the cash-strapped frontier labs are themselves still hiring human "Account Associates" and "Android Engineers" instead of saving those salaries using their own AI capabilities.
https://openai.com/careers/search/
Tao actually pictures the AI as having hacked the FDA, so as to pass the test despite the drug being lethal.
Picking out that one example and misrepresenting it on top of it indeed clearly is driven by an agenda. Or mere stupidity.
The central point in all of this is the misguided idea of "letting AI think for us". An over-generalization feeding into the society-wide problem of self-infantilization.
>to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path
why is it so hard to accept that a brilliant mathematician can have the same human flaws as anyone else i.e. ego, arrogance, jealousy and competitiveness?
Instead of bemoaning his specific and clumsy example, take the gift of this moment and don’t forget that he’s smart, confident, and usually wrong or completely ignorant outside of his own narrow field.
Don’t let the Gell-Mann Amnesia take you.
I think logic is squarely in his field, and even with simple logic, proposing the importance of second order effects on people who are dying is not tenable. Personally, knowing that he is indeed one of the smartest people around, and definitely much smarter than me, I cannot but be a bit worried that going with this example indicates traces of panic in the face of a shifting world. He’s smart but human, and maybe these weaknesses are not to bemoan but to celebrate in our current context.
1 reply →
The conflations about his example ("thought experiment") is twofold, the AI itself confounds the ethical intuition so it is wrong directly compare against what real medicine today does; furthermore, the very admission that real medicine in fact operates through unknown risks (e.g. Jannsen vaccine recall) is different than having a scientific standard that it should not have to be that way at least as minimally as possible.
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
There are probably more mis-aligned humans than mis-algined AI's today.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
I don’t work in drug discovery, but isn’t this thought experiment skipping a lot of steps? My impression is AI is accelerating a lot of the preclinical screening and design work, but that’s pretty far from establishing “tremendously improved outcomes for patients” isn’t it? That’s still really difficult to verify, are people getting real acceleration of this part of the work with AI?
I think everyone agrees that it's totally fine if a drug came about due to AI.
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
> I would say model validation is one of the hardest things to do
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
It's ironic that ML researchers don't fully understand why their models behave the way they do, yet continue to make progress using benchmarks to guide the efforts. Human understanding is important but whether and to what extent it's necessary seems to be a separate question, and the answer may depend on the nature of the field.
> Human understanding is important but whether and to what extent it's necessary seems to be a separate question
We are about to find out the answer soon enough, probably from the in vitro results of the mathematicians currently on the chopping block.
I submit that human understanding is overrated, and many attempts to elevate it in the wake of AI is mediated by protectionism masquerading as virtue and "deep".
What matters more is that what will happen to us when we find the answer. That's what Tao et al. are more worried about.
Traditional way of doing science means we (governments, NIH, NSF, and even private corporations) fund activities like asking seemingly unimportant questions, spending years running experiments on such hypothesis, publishing, reviewing, talking about results, reproducing results and such. We all agree that these are beneficial to us as a whole (Hacker news crowd might disagree). When we understand a process, we can apply it to a different problem and produce something useful. Euler developed a process to answer a whimsical question about walking in a town crossing 7 bridges only once. Now graph theory is applied everywhere.
We obviously have failed to stop OpenAI from dumping "solutions" to hundreds of problems. So going forward, instead of testing hypothesis and talking about results, mathematicians will be forced to read through AI slop and detect what's useful and what's wrong. Maybe it will improve our understanding, but someone has to fund that activity. Will NSF, NIH, or OpenAI for that matter, do that?
In medicine, or drug discovery more generally, there's FAR too little empirical data for training of ML generally. See OpenAdmet. The kickback to "oh but yeah cancer" is currently just hype. DeepMind has pivoted to this realm but has no demonstrable improvements. For the typical phase 1-2-3 pipeline of drugs, stretching many years, there's not yet any demonstrable improvements.
I think your perspective is limited- in cancer, we have copious genomic information that informs treatment (including clinical trials where treatment is determined by AI).
Serious question asking for a serious answer: would you say any of this so confidently if “drug discovery” is the next “mathematics”?
It's more nuanced than that: Tao is not saying people should be denied drugs, but ideally, the mechanisms of why they work would also be understood.
We don't really have that for many present day medicines. We know they work, statistically speaking. But we don't know for all of them how they work, what the mechanism is. For some drugs this lags the discovery of the drug itself (historically: all drugs, but in the last 100 years fewer of them). For some drugs it simply hasn't happened.
https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...
And for some it may never happen...
As usual people that are smart think they can be masters of other fields for which they know nothing.
Claiming that a field medalist mathematician is too smart is peak hacker news.
Who's claiming to be a master of another field?
The effect of the AI-created drug is saving someone's life.
The effect of the AI-created proof is a mathematician abandoning years of research, losing grants, awards, ruining their career, etc.
The only difference here is our emotional reaction!
As a mathematician, there is always the risk that another human mathematician solves the problem before you.
Usually peopla have research lines that are sets of related problems, you must publish one or two paper per year, it's not possible to wait 10 years to publish a big one, or 30 to retire after admitting defeat.
Do you mean to claim that the years of research mathematicians, or scientists in any other fields, don't lead to saving lives?