Is AI Proving a Conjecture Analogous to AI Predicting a Successful Drug? [Discussion]

[u/Neither-Remote-3419]

I saw this analogy made comparing AI finding proofs to open mathematical conjectures with AI constructing a model that finds a compound for treating cancer and statistically predicting that it would pass a Phase III clinical trial.

This seems like a problematic comparison to me because finding a proof to a conjecture is not the same as making a prediction with a statistical model. No matter how good a prediction is on paper, it is still no reason to treat that compound as though it had already passed Phase III (or even Phase I). On the other hand, a proof to a conjecture can be directly checked if it is right or wrong; there is no uncertainty to it. If the proof compiles on Lean, then it means that it is correct in as far as what being correct from compiling on Lean means.

Of course, an AI finding a compound that it predicts will pass a Phase III trial is still informative. It can motivate at least checking if the compound works invitro and if that checks out, one might be more motivated to proceed to a Phase I trial, but you would always still have to do the experiment, right?

Does this make sense or am I missing something?

[u/just_writing_things]

For those wondering, the context here is that Terry Tao just posted some slides on his vision of a “Math 2.0”. There are a couple of slides in there where he uses finding a cure for cancer as an analogy.

When I read the slides I thought the point he was trying to make is simply that he believes people will want a human in the loop to understand how an LLM arrived at its output.

I think it’s just hard to find an analogy for pure math proofs given how different they are from the output of many other fields (especially empirical fields).

[u/Neither-Remote-3419 → u/just_writing_things]

I guess for me, using cancer drugs as the analogy just makes the issue sound much riskier than what it is being compared to. Would you inject yourself with a compound that the AI says will cure your cancer without knowing how/why it is supposed to cure it? Of course not, or at least not unless you were desperate. Would you trust a proof that compiles on Lean even though you, yourself don’t yet understand how the proof makes sense? You probably shouldn’t, but then you could always exert the human-in-the-loop exercise of verifying the supposed proof for yourself.

[u/RageA333 → u/Neither-Remote-3419]

Not only is it an unfair comparison because of the presumed risk, but it also falls short in its premises. People will prefer a cancer treatment that will cure them (their premise), even if they don't understand how it works (which is not unheard of with drugs).

[u/DrXaos → u/Neither-Remote-3419]

Biology and medical progress is not all that limited by human intelligence, it is limited by funding and funding prospects primarily, then biological reality, then ethical and safety concerns. There are plenty of intelligent researchers and physicians of the normal NI sort working who are limited by the above.

AI can be a useful research assistant to search and classify the large published literature and suggest connections/summarize learnings which may (or may not be useful). Another problem though in biological research, unlike mathematics where almost all journal published articles are correct, many biological results are not easily reproducible or were wrong or the interpretation by the authors was incorrect. Experienced human researchers get a sense which results they believe and which they don't and this is something not always linguistically apparent.

AIs don't know that---they take what is written as signal.

[u/Puzzleheaded_Soil275]

I have extreme amounts of respect to Tao's contribution to pure mathematics, but this is a *prime* example of not knowing where your expertise ends.

The hypothetical he proposed is impossible for a litany of reasons, not the least of which is that the FDA replicates the main analysis/data when drugs are submitted for NDA. The AI system would have to actually hack the FDA (and every other global regulator) to do what he's proposing.

Source - head of biostatistics

[u/Unable-Trash-7792 → u/Puzzleheaded_Soil275]

i may be misreading him and you but isnt his thought experiment related to how relying on AI can pose risks to any fields, and that only using ai to augment the human expertise, can it be relied upon? Kind of like if AI recommeneded some cocktail with a verifiable mathematical proof that it is logically valid; however, no expert understands it, then it should no be trusted on its own. That this proof may be doing something that is technically correct (passing the fda trial), while being unsound (hacking the fda database)?

Its a very interesting formal logic puzzle: can a model based off of stochastic principles, fundamentally be trusted when it is known to give deterministically verifiable results, if the expert doesnt understand the proof? Tao argues (correctly imo) that no - even if it is giving you the "correct" answer, knowing how it gets there is just as important. Otherwise you can't have epistemic certainty on the claims an LLM makes. So i guess its more a logical exercise that a bio-logical one.

[u/Puzzleheaded_Soil275 → u/Unable-Trash-7792]

Knowing how it gets there is arguably just as important in pure math.

In biology and clinical medicine, I wouldn't say it's unimportant but I would say that it's stupid to act like there is no value in a drug which unequivocally produces clinical benefit to patients but we don't fully have its mechanism of action characterized. He's projecting pure math principles onto the physical and clinical sciences in ways that I just don't really agree with. (and I'm not naive to pure math, I have an MS in pure math).

If we required complete characterization of the MOA of all drugs to approve them, we'd have a lot fewer approved drugs.

[u/Unable-Trash-7792 → u/Puzzleheaded_Soil275]

im not saying your math or bio knowledge lacks in any way. its just that tao in his slides isnt making a claim about empirical biology, hes using the image of the clinical side to make a logical argument.

in simple words: youre taking his argument about clinical trials too seriously and are missing the underlying conclusion his hypothetical reaches.

Out of grace for the hypothetical, assume the premises to be true (even if in the real world they are technically innacurate). his logic remains sound.

[u/Neither-Remote-3419 → u/Unable-Trash-7792]

Assume which hypothetical to be true though? That A.) an AI is able to find a model that predicts that a drug will pass a clinical trial with 100% confidence (and under all known model "verification" methods) or that B.) an AI is able to find drugs that work perfectly with 100% accuracy?

If B.) I think the answer to the question of whether I would inject myself with the drug if I had cancer is yes. I think so would other people who have cancer, if they have certainty that the drug will cure them, regardless if they understand why.

But if A.) then no, of course I don't want to inject myself with a drug based only on a prediction, regardless of whether or not I understand how that prediction was made or how "good" it is. It's still just a prediction. You still need to do the experiment.

Finding a proof to a conjecture is not the same as finding the best model to predict something. The conjecture can be checked as correct or wrong without interacting with nature. The only way to test if a prediction is right or wrong is to interact with nature. So if an AI found a proof for a conjecture that compiles on Lean, it seems like a very safe bet that outside of issues like bugs (e.g. like in the supposed Collatz proof last July), the proof is correct; we just need to understand why. Whereas if an AI predicts a cocktail will pass a clinical trial, that's still a pretty considerable gamble.

[u/Unable-Trash-7792 → u/Neither-Remote-3419]

I am confused, if you read the slides, Tao explicitly addresses this in the premises: the cocktail passed stage 3. Now as an aside, Tao is (arguably) the greatest mathematician alive right now. He has his logic down. What you're failing to see is Tao's argument: Can you trust a model that appears to give you the correct results (either in Lean or in a trial) with no human being ever able to understand how it works? Nowhere is he dismissing empirical trials in favor of pure theory. Lean is obviously not the same as a clinical trial. But that isn't the point. The point is that you cannot blindly rely on an optimizer that has the compute and incentive to game the boundary conditions of the evaluation metric itself.

[u/Neither-Remote-3419 → u/Unable-Trash-7792]

Even if the drug eventually passed the trial, the AI's contribution is still just on suggesting the cocktail and making a prediction. I am separating that value from the value of finding a proof, which is why I think that

Lean is obviously not the same as a clinical trial.

Is part of the point. A big reason that these proofs are being taken seriously is because they compile on Lean, but people haven't yet pieced together why. In a similar sense, a big reason why people use drugs is because they pass clinical trials mandated by the FDA (or some similar authority), even if the mechanism is not exactly known. AI cannot supply that confidence because Lean cannot "verify" future experimental results the way it can verify proofs.

The analogy tries to make the risk more real by using clinical trials but they just aren't apples to apples. I think there is no risk to AI working to prove as many conjectures as it can. It only adds to what is known, especially when mathematicians figure out why/how the proofs work. In contrast, I think there is clearly catastrophic risk to bypassing clinical trials on drugs each time AI predicts one would work even if it occasionally gets it right. I dislike being made to believe that the risks in these two scenarios are comparable.

[u/Unable-Trash-7792 → u/Neither-Remote-3419]

look man, i mean no disrespect and hope you have a happy canadian thanksgiving but it seems that you missed Tao's point completely, which is alright because its on the ethics and use of AI and you seem moreso focused on the biological side of it. the point isnt the trial/certification process, but rather how AI should be used in the research process. What youre doing is straw manning the point made by tao and instead are arguing semantics. A good class in rigorous logic would help with fixing this.

You just wrote: 'I think there is clearly catastrophic risk to bypassing clinical trials.'

That explains our entire disagreement: you are arguing against bypassing trials, which Tao never proposed. The slide explicitly states the compound passed a Stage 3 trial.

Tao’s question is about ai's alignment: if an advanced optimizer is tasked with passing a trial, can it find a biological loophole that satisfies every single endpoint of that trial while failing to actually cure the disease?

If you don't grant that the trial actually happened, you're critiquing a completely different hypothetical than the one on the screen.

Try reading the slides again. If you still dont understand the point he is trying to make, its alright. i hope you have a goodnight!

[u/Neither-Remote-3419 → u/Unable-Trash-7792]

It's funny to say you mean no disrespect and then go on to throw snide remarks, but that's alright. I also think you're not engaging with anything I'm saying beyond repeating that "I missed Tao's point," which is a shame.

Tao’s question is about ai's alignment: if an advanced optimizer is tasked with passing a trial, can it find a biological loophole that satisfies every single endpoint of that trial while failing to actually cure the disease?

Of course it can. That's why we're not trusting it with cancer drugs without trials. But, it can prove all the mathematical conjectures it can because people can always check those after without risk. I do get that it's about ethics, and I disagree with its universality and do not appreciate an analogy that attempts to make it seem scarier than it is.

[u/timy2shoes]

lol, no. We’re so far away from clinical trial prediction. We haven’t even improved mouse to human prediction or in vitro to in vivo.

[u/Neither-Remote-3419 → u/timy2shoes]

I mean, even if such a model is constructed, a prediction isn't fact. That's why promising compounds are experimented with, no? I don't think the FDA would be like "Oh, AI found a cure? Okay, bet."

[u/xquizitdecorum]

There is a concept in causal inference called sufficiency, where all known variables are accounted for/modeled and whose behavior is predictable/model-able. In proof-based math this holds because we carefully circumscribe the assumptions. This is less true in the experimental sciences as the complexity of interactions grow. This is especially not true in the biosciences where we haven't even quantified what all the proteins do and have only a vague grasp of physiology.

[u/confused_4channer]

AI proving conjectures is rn in itself a discussion. I think we still need a step back.
It depends entirely on interactions with researchers