Comment by n_e
4 days ago
Not sure if it would work for HN / how it could be adapted to HN, but something I noticed on opensource projects, is that once they hit a hurdle, submitters of low quality AI-written PRs don't try to solve it and go elsewhere.
For example, in one project, PRs have to be submitted to the "next" branch and not the default branch. This is written in the CONTRIBUTING.md file, which is linked in the PR template, with the mention that PRs that don't respect that will be close. Most if not all submitters of low-quality PRs don't do anything once their initial PR is closed.
Pretty bummed about that as I just submitted a show HN I'm pretty happy about (it solves an annoying problem I had for years, which I know many people have) and I was looking forward to talk about it (https://news.ycombinator.com/item?id=47050872)
Back when I ran a WoW guild, the first sentence in our recruitment post emphasized the importance of reading the whole post (because the way to access the application form was to click the only smiley in the post, and this detail was mentioned in the last paragraph).
Most people did not read the post, which was immediately evident from how they posted their application by copy-pasting and editing an application posted by someone else before them.
> Back when I ran a WoW guild, the first sentence in our recruitment post emphasized the importance of reading the whole post (because the way to access the application form was to click the only smiley in the post, and this detail was mentioned in the last paragraph).
I mentioned the "no brown M&Ms rule" in a recent comment, and someone pointed out to me that this is more likely to by adhered to by an LLM - humans might miss a single line in a three pages of text, but the LLM won't.
I am starting to think that a better approach might be to move to mailing lists; this means that valuable drive-by PRs (like I did in the past) are going to be unintended victims by preventing all drive-by PRs, because the friction is too high. Submitted needs to have an email address, make sure it isn't marked as spam, sign up to the mailing list, email their PR, wait for a response, etc.
The upside is that projects can start marking that email address as spam if it submits AI slop. The downside is that actually valuable drive-by PRs will be a thing of the past.
I wonder if you could take advantage of the fact that the LLM is more likely to follow instructions that humans might miss. For example include instructions somewhere in the repo that says you must use a certain phrase in all pull requests, and then you just check the PR for that phrase.
Or maybe require the PR to contain something that is generated by running code, which the LLM may not be able to do without some effort on the user's part.
> For example, in one project, PRs have to be submitted to the "next" branch and not the default branch. This is written in the CONTRIBUTING.md file, which is linked in the PR template, with the mention that PRs that don't respect that will be close. Most if not all submitters of low-quality PRs don't do anything once their initial PR is closed.
Few things in life are as reliable and trustworthy as the laziness of others.