Comment by qup
8 hours ago
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No. I'm gonna die, my man.
This guy might have the highest IQ on the planet, but it's clear he hasn't spent much time around average people.
This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
Will I take a cancer cure that no human understands? Yes. And so will billions of others. Just needs to cure cancer, that's nearly the only requirement.
And to be fair - that how medicine works today. We don’t really understand what goes into our body, so that’s why we do clinical trials. And sometimes we discover adverse side effects years and decades after a drug has been released.
Yes, and that isn't the model's fault any more than it would be for a human.
That's what the trials and all are for.
Who cares about understanding it, in the face of efficacy?
I want to understand everything; I think AI will help that happen, not hinder it.
All the arguments about the future mathematicians are imagined, and emotional.
Yep, I think a lot of them are grieving the loss of their identity. Quite understandable given how much of their life force they've poured into math, especially at the level of that Tao is at.
It makes me think of the book Finite and Infinite Games. It provides a perspective that work is just one role that we _choose_ to assume in our life. Realizing that we can choose other roles and move in and out of them freely has helped me alot with big changes (career and otherwise) in my life.
1 reply →
> All the arguments about the future mathematicians are imagined, and emotional.
Even if you don't like, at this moment, AI cannot replace mathematicians.
Indeed - we don't know all of the mechanisms of Tylenol, yet we know it works, and is safe when taken as directed.
Ironically this is one of the worst counterexamples he could have selected.
If you read all the questions asked by Terrance you would have understood what he is thought behind that question "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?"
He is not talking about a cure that works and that no one understands, he is talking about an AI making its way out of the trial just to get to phase 3...
5 replies →
Not only that, but also those clinical trials often have low sample sizes. Trials involving only a few hundred patients, or even fewer, are pretty common.
not just clinical trials- there is an enormous amount of in-vivo model study before the molecules go into people.
In vivo is already molecules in body. Do you mean in vitro?
1 reply →
Agreed. IIRC the mechanism for anesthesia in unknown to this day. In fact, medicine as a whole is very pragmatic and time and time again prefers using techniques with unknown mechanisms but proven results than waiting for a reasonable scientific explanation of what’s going on.
As logic would dictate--particularly for people who aren't the highest IQ guy on the planet.
Half of people are below average! They understand nothing at the level that Tao means. Literally nothing.
Perhaps I'm in the lower half but I do not understand what point you're trying to make. Are you saying that as some higher plane of understanding Tao is actually correct? My thinking is that a simple counterexample would be sufficient - i.e. are there medications that people use for which the mechanism is not understood? There are many such examples as others have given in this thread.
9 replies →
I don't expect Tao to be well versed in medicine, but yeah, that slide is detached from clinical practice.
Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.
I don't know how to word this in a way that won't get me into a pointless semantics argument, but the word Cure in the slides is clearly used to mean "substance intended to be (but not necessarily confirmed to be) a cure". If you think it's badly worded that's fine, but ostensibly the point here's not to "win" the argument game, but to get his intended point and engage with that.
No, it doesn't mean "intended to be". "While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial".
My own pointless semantic argument: we typically don't use the word cure when we talk about cancer. Instead, we use the term "complete remission". Many people who were thought to be cured then developed cancer decades later that was genetically derived from a small remaining population of cancer cells that were not eliminated in the original "cure". The word is a shibboleth for not being familiar with cancer medicine and treatment.
The above commenter has the same allegory- most “users” of math don’t care about the body of work behind it; only the consequence of it being proved is the fact that matters.
But the slide says it passes a stage 3 trial. Surely many dying patients would kill for the opportunity to take the drug.
Many people would, many wouldn’t. The comment I’m responding to casts the decision as so uncontroversial that even asking the question is inhererently damning.
1 reply →
Beat me to it. It is sad to see that a lot of the people commenting lack good reading comprehension. Or maybe it is that they just want to be dismissive of something they don't like, or just for the sake of it.
I would totally take the mysterious drug if it was clinically tested and shown to be reasonably effective. But that is not the premises presented here.
Wrong: the slide says "the cocktail indeed passes a stage 3 trial." Be careful before you accuse others of bad reading comprehension.
Mathematicians inhibit and explore a world that is completely made up and exists only in abstract.
I wonder if this makes them slowly get disconnected from the lived realities of billions of ordinary people.
How could a theoretical mathematician not be massively disconnected from the reality of life for the ordinary person? We usually celebrate their quirks until they come into conflict like this.
I agree, how could he --also being so smart.
It's a good reason not to take his advice about the real world--like how to manage AI's trajectory.
The cancer example illustrates this gap perfectly. He thoughtfully crafted the point, and defeated himself in argument.
Unless it was a move?
When thinking about math, Tao is in another world unfamiliar to us. But, he grew up in our world and is a professor in our world and if you listen to him, seems like a pretty normal guy. I think he just made a bad analogy here.
If true, this would be observed at least partially in software engineers as well, since what we deal with is kind of made up as well, but I don't see it.
> This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.
You're discussing the medicinal chemistry process, which is very different to understanding the molecular mechanism by which a drug operates (Tao's point).
>This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
he isnt saying that person who puts stuff into their body should understand it
it is that someone (expert) should understand it to the point he can vouch for it
False. Experts don’t understand many drugs and still prescribe it. And it’s a good thing.
Is that a satisfactory state of affairs? "Experts" also prescribed thalidomide for years before embarking on a decades-long scramble to unravel its teratogenic properties. "Experts" also initially prescribed ivermectin and hydroxychloroquine for Covid-19. While empiricism is an important part of medical sciences, medicine is not a purely empirical subject because pure empiricism is insufficient in the long run.
The software analog of settling merely for "it seems to work" would be if someone were to hand you a bare binary whose behavior must be inferred entirely through black box testing, without source code, architecture diagrams, or any other documentation. Sure, users can try running it in a pinch, which is the case for various medications, but it's surely possible to do better in the long run.
Nobody understands the mechanism behind general anesthesia, yet there’s a medical speciality dedicated to practicing it and it’s done every single day for a wide variety of procedures. We have figured out how to use general anesthesia in a relatively safe way, but nobody understands why it works.
Doctors prescribe accutane
> This argument is so silly against reality
Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.
We don’t even know how acitaminophen works, thousands get their livers destroyed every year from it when alternatives exist, and we are yet to ban it. So how is all of the new reality any different?
Out of curiosity, what alternatives exist for moderate pain / fever management? I imagine NSAIDs are even worse in terms of adverse effects?
Has the OP text been changed?
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Now:
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism behind this cure?
I would agree with the second, not necessarily the first.
Yes, I believe somebody pointed out to him (maybe here?) the deep flaws and he attempted to strengthen it. In fact, it looks like the changes were done by Claude (!!!).
The old text:
"""Suppose an advanced AI is prompted to “find a cure for cancer that passes a stage 3 clinical trial”. After a large amount of compute, it produces a cocktail of previously unknown chemicals which its mathematical model predicts, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial. Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process? Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure? Or a human mathematician who understands the mathematical model used to locate the cocktail?"""
The new text:
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial. Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
I am not going to pay attention to Tao's opinions on this from now because I don't really have faith that he's even writing this or that he understands why you can't make a lean cert for a drug discovery (yet!?)
Funny enough, it is a good argument with just a sprinkle of reality for context: Assume effective alternatives to this cocktail exist. Assume you had access to these effective alternatives. Can this be combined with other drugs? Surgery? You say you just need to cure it. You must consider the deeper mechanisms at play to consider all the possible options for living.
I will take the safest route to a guaranteed cure.
In the absence of a guaranteed cure, I will probably take the option or combination of options most likely to cure me that I can afford.
Whether the AI understands and endorses it vs a human understands and endorses it, is pretty much totally irrelevant to me.
The AI will have a track record at this point for me to make a viable comparison.
Also, why not both options, assuming I can afford it and they're compatible.
I believe AI will invent new treatments in cases where there aren't options, those people will be cured, and using AI medical techniques will become obvious.
I think it is more disconnected in thinking people want to understand the mechanism. As a stage IV cancer patient, I’ve met many people who decide not to continue treatment with approved drugs due to side effects. The notion that every cancer patient will do everything possible to stay alive isn’t true. Most people have a breaking point.
He’s making the same argument I’ve heard software developers make for the past 3 years. Lawyers are saying similar types of things. Knowledge work isn’t as special as we all thought it was. Now, it’s Terence Tao’s turn to go through the same motions we’ve had to go through over the past few years.
I never went through them.
I did not think my ability to write code was special, and I was always thrilled the clankers might do it for me.
But yes, I agree. And many are in line behind Tao, soon to have their turn.
Plumbers are feeling pretty good right now.
Until we design fittings and fixtures that just snap together. This has already happened to some extent in the construction trades. It doesn't outright replace labor but it reduces the labor content of trade work. It's a myth that automation can't replace labor.
> Plumbers are feeling pretty good right now.
Until their job market is flooded with all the career switchers.
The problem is you won't be faced with a binary choice of certain death or a single unknown AI cure: You will have a choice between several established protocols AND several unknown AI drugs. You doing a dice roll or do you want to make an educated decision?
The AI made a more educated decision than I'll ever be able to make.
And to frame one as a dice-roll and another as not-a-dice-roll is rich.
TBF, math is the most ivory-tower of the sciences in that every claim in the field has to be fully understood by proof.
> t's clear he hasn't spent much time around average people.
You haven't spent too much time around academics then. These people more often than not have not spend a single minute talking to the layman.
You agree with my thought, but you think I'm naive to believe it?
Would you take an "AI-designed" airplane where no human understands its physics?
Would you drive on an "AI-designed" bridge unvetted by expert engineers?
Maybe. Maybe not.
Thing is there isn't a "principle" from we won't use AI designed/implemented things. It just depends on the cost/benefit/risk balance, and the risk part is largely subjective (because we don't have enough data, and if we avoid using AI before we have enough data, then we won't have data for a long time).
There's a difference between "no one understands it" and "no one has tested and validated it's safety". You don't need to fully understand the mechanics of something to test it and validate it. Plenty of things in the real world work this way already.
Will I die if I don't get on the airplane?
If so, yes, I'll get on it.
It doesn't even need to cure cancer, just prevent it to some unknown degree with unknown side-effects. As long as they want people to take it, they will.
Congratulations, you’re patient zero for an unstoppable virus
Humans already did that
I think his argument is more "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?" So an AI agent could game (lack of a better word) the process and produce something that may cure cancer, but misses something else.
Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Do you think the current clinical trial process has not been gamed close to the edge of uselessness ?
Do you know about the fiasco with the Alzheimer’s drug that has collectively cost humanity hundreds of billions?
I don’t see how an indecipherable AI based cure can be more misaligned.
> Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Plenty of people will reject all AI things, fully agree.
But you mischaracterize COVID history. People were not again a COVID vaccine, by and large. They were against a rushed vaccine. They were against a vaccine without human trials. They were reluctant to guinea pig mRNA vaccines. And they opposed forced/required vaccinations.
Those are all reasonable takes. And depending on the reasoning, I can even see being anti-AI. I think a lot of spiritual people will land there, and that belief system makes their choice logical.
And if you've seen any of the Fauci stuff recently, you'll know it wasn't dumb at all.
You're right on -- the point is an alignment point in a talk that only uses "alignment" in a completely different sense than it usually is, which is incongruent at best. It's gonna lead to a way bigger backlash among laypeople/policy makers/non-expert stakeholders than the rest of the talk will lead to new growth, combined :(
Slide 22 (from a satirical social media post) explains Tao's opinion, IMO:
> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.
He mentioned this hypothetical cure has passed human trials, in his argument
To be precise, he said:
> ... does indeed manage to pass a stage 3 trial.
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
This is the take I don’t see enough and it’s blisteringly obvious to me and it’s unclear why that it’s not more obvious.
The amount of things people understand about the world is effectively zero, yet they rely on them all the time.
There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane
I think the answer is that it’s all about the rate of change
Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.
If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable
The cancer thing is just a contrived analogy. His real concern is having himself and his mathematician friends replaced by AI. Appealing to someone's health is an easier sell and then it can normalize the relationship of medical priests, and math priests, etc.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.
I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.
I'm still a fan of Terence Tao, I'm really sad to witness what appears to be the 5 stages of grief.
The problem with his argument is that he put the cart before the horse. He presupposes we've found and validated a cancer cure, but at that point any rational person would accept said cure. The moral quandary is how you are going to validate it against all the other possible candidates, when you don't have a coherent rationale for it to work.
What makes that quandary the moral variety?
Not understanding a mechanism doesn't make me, or the mechanism, good or evil.
We just treat it like all the other cures right?
> but it's clear he hasn't spent much time around average people.
Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.
I own a bar, so...yes, I have. More than my fair share.
And yes, that made the cocktail distinction easy for me--it was right in my area of expertise, to rule out one variety.
If the smartest guy on the planet forms his own argument, don't you think it oughta be a good one? Pretty convincing?
It's science (understanding) before engineering (efficacy).
The most common Science™ failure mode.
I agree, if you mean he's prioritizing that.
But since when do we call for a stop to figuring out new things? We don't want a cancer cure unless we understand it?
N people should die because...math is cool for humans? Or?
What if it takes your five years to live down to two in horrible agony?
Then it won't be a cure for cancer
Tao called it a cure, see the thing I quoted.
I think this is just an embarrasingly bad faith / low effort way of engaging with the argument.
"its model predicts [...] will kill cancer cells"
5 replies →
I mean stuff like this happens in trials without AI too.