Comment by Aransentin

3 years ago

Some napkin math: Given rectal cancer's rate of survival of 67%, and the small size of the study (18 people), you should see similar results due to random chance every 1350th study.

A cursory search on clinicaltrials.gov and I can find 7131 cancer studies started in 2021 alone. It's therefore not unreasonable for this one to be just a random fluke.

This analysis is mindless and inappropriate. If you care about this at all do yourself a favor and just read the study.

https://www.nejm.org/doi/full/10.1056/NEJMoa2201445

Otherwise redo your napkin math and cursory search to answer, specifically, whether all these cancers disappearing within weeks of dostarlimab treatment could be a fluke. And do not compare this to "rectal cancer's rate of survival", which is irrelevant and a completely different set of (parametrized) statistics, and also do not compare it to the total number of "cancer studies", which was an arbitrary choice and yielded this meaningless conclusion. Even if this kind of analysis was useful, why did you compare against the number of cancer studies, rather than rectal cancer (1910), or dostarlimab (41), or studies with the same staging and genetic pathology? It's meaningless.

I don't believe you're qualified to tell anyone about the significance of this study, and much less dismiss it as a fluke.

  • The point of that napkin math is not to get an exact prediction, it's to get a order-of-magnitude estimate to see if this study is different from any other of the large amount of "promising drug cures X" articles that never actually turn out to work.

    I concede that if a quick remission is extremely rare that does change the outcome, though I don't know how common that it.

    Picking 'Total number of "cancer studies"' was not arbitrary. A similar article could have been crated for e.g. "promising drug cures breast cancer" so you need to take all of them into account. Actually probably every study conducted of all high-profile diseases that are likely to wind up on the HN frontpage - in a counterfactual universe we could be discussing a miracle Alzheimer drug or the like.

    • No, you can't actually figure out whether a cancer therapy worked by counting the number of articles published in a given year. You're focused on this "zero-knowledge" approach because you are completely ignorant about this topic, not because it is actually a more powerful approach.

      If you don't even know whether spontaneous remission is rare then you should not be offering an opinion about this. It's not even pertinent because these remissions were clearly not spontaneous, but it should alarm you to have spoken so recklessly about a disease which is essentially fatal. What you're saying amounts to "there's a good chance cancer just goes away on its own". Incidentally, this is also the least amount of math I've ever seen in a napkin math argument.

Pretty sure the base rate you'd want to compare against is either placebo given -> remission or simply spontaneous remission. It appears that spontaneous remission is really quite rare.

> [sponanteous remission of cancer] incidence is roughly one in every 60 000–100 000 cancer patients, but the true figure is unknown (2). Spontaneous regression of colon cancer seems to be particularly rare

https://academic.oup.com/jjco/article/45/1/111/888056

So using 0.0016% and 12 patients (which is what the paper the NYT actually links)

For a ~50% chance of seeing one trial with 12 patients have complete remission you'd expect to see ~43k trails.

(1 - 0.000016)^43000 = 0.502577

I wouldn't discount this study based off of those numbers.

  • If spontaneous remission is super rare it does change the conclusion, yeah.

    (Tangentially I think your stats are a little messed up there - you calculated the expected number of people out of 43000 surviving, not the trials returning a false positive. Assuming 0.0016% that chance is astronomically small, so your argument is stronger.)

This napkin math ignores some very significant circumstances.

People who suffer from rectal cancer usually undergo surgery to remove the primary tumor. But those trial patients weren't operated on, their treatment was non-invasive.

How many rectal cancer sufferers who never undergo a surgery survive? I would bet that it is a lot less than 67 per cent.

  • And this was people with locally advanced rectal cancer. Typically this means the tumor has grown to a considerable size and is already causing symptoms severe enough for people to go to a specialist. I am not a doctor but my understanding is that a placebo in this case would have a 5 year survival rate that is pretty close to 0%.

There should be some kind of award for these kinds of "well actually" comments on HN that lack any kind of intuition for the domain.

Cartoon montage: "By my calculations..." followed by driving a car off a bridge.

Edit: as someone who works in cancer research, I can tell you that your prior for 18/18 locally advanced colorectal cancer patients achieve CRs without surgery should be ~0.

They did not just survive. They had no traces of cancer. That chance is much much lower.

Your basic result stands, but "survival" and "remission" don't necessarily equate. But I agree with your basic point.

If I understand the article correctly they were excluding patients enrolled in chemo and radiation?

If that's what they mean the survival rate wouldn't be 67% so this would imply a 1349 in 1350 chance the treatment is better than the average treatment?

Other commenters have done a fine job expressing why this napkin math is silly and perhaps in isolated cases this is a good thing, OP probably learned something here

But in aggregate these sorts of comments are annoying as hell on every medical article posted to HN. Now you have to hope a sufficient mass of well-informed commenters is here to rebut them. In the best-case, these comments are simply misguided, but in the worst case it becomes a watering hole for all the antivaxxers and conspiracy theorists on this site to gather

If i put a single coin to the vending machine and get 7 cans instead of 0.98, i surely will try a few times more before reaching the conclusion my coins are magical beans. Visibly medical research jump so quick to conclusion it's to the millions of news reader to swallow the clickbaits.