← World & context

The problem with AI citations is that they are all too human

Published here:

Citations have become a battleground for research integrity, with papers being screened and censured for their use of inaccurate citations produced by LLMs. Discussing an experiment on how LLMs cite work in the social sciences, Egor Kholin suggests while their accuracy is increasing, the social nature of citations still makes LLM outputs problematic.

In 2003 two physicists gathered around 196 incorrect citations from scientific articles. They found the most common one was repeated in 78 different articles. Most writers didn’t cite the original work, they cited a secondary work that referenced the original work. By simple calculation they estimated that only 20% of writers had read the original work .[1]

We live in a world where LLMs have “read” almost the entire scientific literature and thus in theory everything they cite. This would suggest LLM citation practices could be more precise than those made by human authors.

An experiment

To compare the difference between LLM and human ways of citations I produced 123 machine-written texts, mostly lectures of about 3000 words, and pulled around 3600 citations out of them. I then tested them against classic theories of citation.

Social science teaches us that the process of citing isn’t just a note of where an idea or finding has been copied from. It is far more complicated and can be seen as a symbolic payment, hedging, or even sometimes a ritual honouring the gods of a given field of research. To borrow a term from Erving Goffman, a citation carries a key, a signal telling you how to read it. Human citations consist of multiple meta-signals, an experienced reader sees big names in a text and understands that it’s a ritual, they don’t need to read this, they see a critics name after a statement and understand that it is a critic by parenthesis, verbs, author’s reputation. Are LLMs then able to replicate these nuances in human citation patterns?

Faulty memories, but fewer “hallucinations”

Current AI instruments are better than most humans at simply getting the reference right. Three years ago 18–55% of LLM citations turned out to be fabricated . Around 99% of references now point to real works, which was confirmed in this study .[2][3]

Here’s one thing that surprised me a lot. Sonnet cites Lacan’s Écrits in the old Sheridan translation. We had another edition on the table. The page numbers differ by exactly 74 in all 9 checks. The machine recalls the pagination of a specific paper book. For example for Weber it got 18 pages out of 19 right, including p.181 for the “iron cage”. The one page error we found was the right words on the wrong page: “ascetic compulsion to save” was placed on p.172, the phrase actually sits in the endnotes on p.278. The machine didn’t hallucinate; but it mirrors the behaviour of a fallible human note takers.

In the same way its quotes are also fallible . Out of 300 quotations with a page number, 165 we were able to trace to the same edition, and three of those were fake. All three were quite a popular thesis dressed up as a quote: “took the monastery out into the world” instead of Parsons’ “strode into the market-place of life, slammed the door of the monastery behind it”. The same thought, just in other words. Humans are often guilty of the same error. “It is not the strongest of the species that survives” is not Darwin, it’s a management professor paraphrasing Darwin in 1963 .[4]

Paying homage with citations

Robert Merton described citation as a currency : a currency for recognition, where the returns compound for those who are already recognised . An experienced reader knows this and skips the ritual bracket . ChatGPT puts a citation under a commonplace in 22% of cases, and in half of them it pays the big name of the topic. For example “(Weber, 1930)” under a sentence that doesn’t attribute anything to Weber. Sonnet does the opposite: 81% of its multi-name brackets support a specific claim. When a human puts in a ritual bracket, you can tell by where it sits. When the machine does this it can be indistinguishable.[5][6][7]

Confusing critics with allies

LLMs therefore exhibit some parallels with human citations, but they also introduce their own pathologies. Bruno Latour developed the idea of citing against someone, when you bring a name into your text not because it supports you, but because you want to argue with it. A human usually knows perfectly well who their rival is, it’s personal.

However, LLMs sometimes mix the critic and the supporter. The machine, for example, cited Gordon Marshall as a critic of Weber, even though Marshall ends his book (Presbyteries and Profits, Clarendon Press, 1980) with a part titled, quite unambiguously, “Weber vindicated”. A similar case came up with a study on church attendance and economic growth , where the model turned the result into a positive link while the original paper reported finding a negative one. Our own generator did this too, which is why we had to build a separate judge for it. This seems to be the place where the machine still struggles with the human dynamics of citation.[8][9]

Yet even this error is not entirely alien. In the texts on which LLMs are trained, a thesis and its critics tend to appear side by side: in the same literature reviews, the same reading lists, often the same paragraph. Names that frequently appear together end up closely associated, so Weber’s critics and his defenders become neighbours. A student reading the debate the night before an exam might make the same slip.

Sources become malleable

Among citations to the works we hold in full text, about 10% of Sonnet’s specific claims and 6% of ChatGPT’s are not in the source. Book and author are real, but the specific claim doesn’t exist in the source. A reader can’t find it without opening the book.

Something similar happens when we took a work which was used in different outputs from the experiment. The work returns in other versions, but it started to support different theses: only 25–29% of the claims came back with it. For a human citing the specific work is a to an extent a memory or representation of what they read, or expect, to read. But for machines it’s built every time from scratch, the name exists, but not the key.

All too human

Why, then, do LLMs cite like us? Most likely because they learned from us. Having been trained on human research papers, they reproduce not only the works they cite, but the ways in which humans cite them. Even their machine-specific errors have human equivalents: the same book cited in support of different claims, a straw man in place of a real opponent, critics and allies confused. These are the errors of a careless reader.

Let’s go back to those two physicists for a second. Only 20% people actually read what they cite, and still, when we read a human text we trust its citations more than we trust the machine’s. Even though the machine has read almost everything it cites. That sounds counterintuitive, but just think about what the human is doing. The researcher who never opened the paper they cited still knows why he put it there. There is an intention. They are paying somebody back, or picking a fight, or they just need something in the bracket so the sentence doesn’t look naked. And the reader knows that too. The machine has read the paper and has no idea why it’s there. So maybe the problem with AI citations was never that they don’t exist. They exist and they are often accurate, but they can’t tell you why they are there.

This article gives the views of the authors, not the position of LSE Impact or the London School of Economics. You are agreeing with our comment policy when you leave a comment.[10]