Comment by gnfargbl

>Are you saying that despite this, these malicious commits made it to production?

Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet.

The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.

It seems that Greg K-H has now released a patch of "the easy reverts" of umn.edu commits... all 190 of them. https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...

The final commit in the reverted list (d656fe49e33df48ee6bc19e871f5862f49895c9e) is originally from 2018-04-30.

EDIT: Not all of the 190 reverted commits are obviously malicious:

https://lore.kernel.org/lkml/20210421092919.2576ce8d@gandalf...

https://lore.kernel.org/lkml/20210421135533.GV8706@quack2.su...

https://lore.kernel.org/lkml/CAMpxmJXn9E7PfRKok7ZyTx0Y+P_q3b...

https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...

What a mess these guys have caused.

They aren't lying, but their methods are still dangerous despite their implying the contrary. Their approach requires perfection on both the submitter and reviewer.

The submitter has to remember to send the "warning, don't apply patch" mail in the short time window between confirmation and merging. What happens if one of the students doing this work gets sick and misses some days of work, withdraws from the program, just completely forgets to send the mail?

What if the reviewer doesn't see the mail in time or it goes to spam?

GKH, in that email thread, did find commits that made it to production; most likely the authors just weren't following up very closely.

> Are the authors lying?

In short, yes. Every attempted defense of them has operated by taking their statements at face value. Every position against them has operated by showing the actual facts.

This may be shocking, but there are some people in this world who rely on other people naively believing their version of events, no matter how much it contradicts the rest of reality.

even if they didn't, they waste the community's time.

I think they are saying that it's possible that some code was branched and used elsewhere, or simply compiled into a running system by a user or developer.

This is one of the commits that went live with "built-in bug" according to Leon:

https://github.com/torvalds/linux/commit/8e949363f017

The particular patches being complained about seem to be subsequent work by someone on the team that wrote that paper, but submitted since the paper was published, ie, followup work.

'race conditions' like this one are inherently dangerous.

Yep. To take a physical-world analogy: Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? Or would that be a go directly to jail sort of an experiment?

I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.

Out of interest, is there any way to have some sort of automated way to test this weak link that is human trust? (I understand how absurd this question is)

It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose.

Edit: NM, see st_goliath response below

https://news.ycombinator.com/item?id=26888538

Such as? Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs?

The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either).

As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing.

What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money.

For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith.

Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code?

Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns.

But what technical measure would you propose that can effectively stop con men?

Force the university to take reponsibility for screening their researchers. i.e. a blanket ban, scorched earth approach punishing the entire university's reputation is a good start.

People want to claim these are lone rogue researchers and good people at the university shouldn't be punished, but this is the only way you can reign in these types of rogues individuals: by getting the collective reputation of the whole university on the line to police their own people. Every action of individual researchers must be assumed to be putting the reputation of the university as a whole on the line. This is the cost of letting individuals operate within the sphere of the university.

Harsh, "over reaction" punishment is the only solution.

The only real fix for this is to improve tooling and/or programming language design to make these kinds of exploits more difficult to slip past maintainers. Lots of folks are working in that space (see recent discussion around Rust), but it's only becoming a priority now that we're seeing the impact of decades of zero consideration for security. It'll take a while to steer this ship into the right direction, and in the meantime the world continues to turn.

The University and researchers involved are now default-banned from submitting.

So yes.

It does raise questions though. Should there be a more formal scrutiny process for less trusted developers? Some kind of background check process?

Runs counter to how open source is ideally written, but for such a core project, perhaps stronger checks are needed.

> Were we under the impression that it's impossible to introduce new (security) bugs in Linux?

I've heard it many times that they're thoroughly reviewed and back doors are very unlikely. So yes, some people were under the impression.

Not really. You can't introduce bugs like this into my companies code base because the code is protected from random people on the internet accessing it. So your first step would be to find an exploitable bug in github, but then you are bypassing peer review as well to get in. (Actually I think we would notice that, but that is more because of a process we happen to have that most don't)