Comment by dataflow

Even as a human, you can still fuck up references.

I submitted a paper with a reference author as Elisio because I couldn’t read my own handwriting. After submitting, I double checked all the references through an LLM. It pointed out that their name was actually Enrique. Yes, you should probably double check your references before submitting, not after.

Point is, I didn’t even trust the LLM at first. But after verifying the mistake, I was embarrassed af. I resubmitted with the fixes before it went live, but ultimately, what’s the difference between “mistake” and “hallucination”?

I do believe this policy is appropriate to deal with the reckless disregard of posting hallucinated references.

It's a conscious decision to not take the time to check your AI output, and instead waste a whole bunch of other people's time letting them essentially do that for you in duplicate.

Feels like that should disqualify you from participation for a bit. Intent or no intent.

The intent to deceive is there. The deception is lying when you submit it that it is a scholarly piece of work in which amongst many other things you know the citations are accurate. This false representation was knowingly and intentionally made at the time of submission.

The citation being incorrect is merely the proof of deception not the (relevant) deception itself.

Fraud is the correct description provided (and this is practically a guarantee) you intended to benefit from the submission of the paper (e.g. by bolstering your resume).

I think (though might well be misunderstanding) that reckless disregard is taken to be an intentional choice but that it does not imply that the outcome itself was intentional. The difference between intentionally doing something that you know for a fact has a high risk of failure but you can't necessarily predict the outcome versus intentionally seeking a particular legally disallowed outcome.

But what LPisGood was saying is that reckless disregard (as opposed to explicit intent) is sufficient to meet the legal bar for fraud.

Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference” for the veracity of the statement asserted.

If you are using AI-hallucinated references in scientific papers then there is some obvious intent to deceive there

> No, it is emphatically not. D Fraud requires intent to deceive.

I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here.

Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature. If you're posting things to arxiv that are not sincere engagements with the literature, you are attempting to deceive.

> followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue"?

This part seemed reasonable too. I'm not in academia, but my understanding is most people writing papers intend for them to be accepted by reputable peer-reviewed venues, but post to arXiv because those venues don't always allow for simple distribution.

If your papers aren't going to be accepted at reputable venues and you posted slop to arXiv before (and they noticed it!), seems reasonable that they only want reputable stuff from you in the future?

Poe's law striking hard.

If automated research is the future, it has to be research, not making stuff up.

Which of those two does "hallucinated references" fit into?

There are no deadlines for journal submissions. Even if you felt you were running close to your revisions being due, an email to an editor will probably fix this for you. And what you described is still negligent, not verifying the garbage output bot did not in fact output garbage.

Your constructed hypothetical makes it even worse. If there are 2+ people in this scenario who have good intentions, this should especially never happen. When you sign your name on a paper, you are nonetheless vouching for everything written in it, including the things you didn't personally write. You should absolutely be checking every single reference your co-author included and verifying that it says what your co-author claims it says. This is something you should have been doing completely independent of LLMs existing. This is something you're publishing publicly, something that may be associated with you and your career for the rest of your life, it is insanely negligent to not even read and verify what your co-author is adding.

In other words: all it needs for your paper to have fraud is for your lab partner to add fraud to your paper.

I'm not seeing the problem here. The only problem is that your lab partner should be banned and not you. But being incentivised to check your co-author's work before submission isn't a bad thing.

You’re confusing the issue here by saying it’s not your fault, it’s your lab partner’s. We’re talking about why your lab partner did something wrong. You can assign blame for the wrong thing separately.

The citation is part of the substance of the paper. If you YOLOed in a citation without checking it, seems justified to suspect that you may have YOLOed in some data, or some analysis, or maybe even the conclusion.

This is like saying lawyers should be allowed to submit AI-hallucinated case references or quotations in court documents. Because by your logic, that, too, should be perfectly acceptable. Yet is not, for hopefully obvious reasons. Why exactly should scientific research be any different? If your paper contains hallucinated references, we can't verify your assertions in the paper, and therefore must question the paper as a whole.

You seem incredibly upset you can't get away with fraud and that people are calling it fraud.

The lack of understanding that you are responsible for the content you create, no matter what tools you use, is what's astounding.

if you're not checking citations in the paper youre publishing AND trusting a non SOTA, hallucination prone ai model to come up with sources for it, its probably for the best of everyone that this paper isn't published.

yes there will be rare exceptions but in general i feel like this is a really good addition.

Why would you ask the ai to find a citation you know exists? Just reach for that citation.

if an llm does the work, you did not write it or research it, the llm did. you have no business crediting yourself as an author.

if someone writes a paper and an entirely different person takes credit for it without even bothering to check if the actual writer just made shit up, they deserve a lifetime ban. seems like a year is a very light punishment.

Yes, having AI write something and not checking it yourself is sure to lead to hallucination, hence, it is a fraudulent way to write.

>Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists

Assumptions:

1. The entire document is loaded into an AI editor

2. The researcher is asking an AI editor to work on his references

3. The researcher has not checked his own references.

This could be avoided at 1, 2 or 3. But even just 1 implies that the researcher knows that they have a hot potato and might critically fuck up and lose all credibility. Being in that scenario and committing to 2 and 3 is at least extreme negligence.

If you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.

Verifying that the reference you cite actually exists is the absolute minimum standard for academic work. It is not optional, not something to skip because of a deadline, and not something to outsource blindly to hallucination-prone AI.

If someone cannot meet that bar, they have no business publishing research papers. I have written academic papers myself, and I find it astonishing that people are trying to justify this as if it were some understandable workflow mistake. At that point it is simply slop with academic formatting. Post it on a blog or somewhere else, but do not put it into the scientific record.

A one-year ban is not a lifetime ban. Maybe six months would also have been enough, but the author can use that time to think about whether they should verify references next time — and to manually check every other citation.

I would not necessarily go as far as calling it fraud, but if you cannot even verify that the reference you are citing actually exists, you are not ready to publish research papers.

Deadlines are not an excuse here. Checking whether a cited book, paper, or passage exists is the absolute minimum standard for scientific work, not an optional extra. I have written academic papers myself, and I find it astonishing that people are trying to justify this as if it were some understandable workflow mistake. At that point it is simply slop with academic formatting.

A one-year ban is not a lifetime ban. Maybe six months would also have been enough, but the point is that the author gets time to think about whether they should verify references next time. They can also use that time to manually check every other citation.

> Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing?

Indeed I cannot. If you do that, you are not, in fact, an honest researcher. You're a lazy hack.

> One hallucinated citation does not in any way imply anyone is being left behind.

The parent said “setting” others behind, which refers to lost time.

Being “left” behind implies a degraded trajectory, which is defined not by time lost, but by the final destination.

Different but related things (e.g. lost time can indeed affect your final destination, for instance, after growing old correcting a scourge of hallucinated citations - which should have been table stakes all along).

No. It's fraud.

You clearly misunderstand. You cite a work in your paper because you have read that work, and build upon it or want to refer to it to back up a specific claim. Generating references is fraud period, because you are implying that you have read a work when in fact you just asked an AI "please insert some reference-shaped text here" to make it look like a proper paper. It is sadly not a necessary, but certainly a VERY sufficient, reason to conclude a paper is fraudulent.