Causal overclaiming is common, rising over time, and difficult to avoid
Published here:
Academic scholarship is meant to proportion claims to evidence, yet scholars can oversell their findings. Calvin Isch discusses the findings of a large language model analysis of 194,631 observational social science articles and finds causal overclaiming was common, increasing over time, and prevalent even in top journals. Considering the causes for overclaiming he suggests what editors and authors can do about it.
It’s a January morning in 1995, at a bank in Pittsburgh. Everything is running smoothly. Then two men burst through the door.
“Hands up! This is a robbery.”
The branch manager freezes. “Okay…let’s stay calm. You can have the money.” Then she notices something odd. These robbers aren’t disguised. Their faces are completely visible. And wet. They’re blinking furiously. “Are you alright?” the manager asks as she slides money across the counter.
“Fine.” the bigger one grunts, still blinking. He grabs the cash, and they back out the door, faces glistening and exposed to the security cameras.
Days later, police arrest Clifton Johnson. His partner, McArthur Wheeler, takes a few months to track down, but the security footage makes identification trivial.
When detectives show Wheeler the surveillance photos, he gasps in disbelief. “ But I wore the lemon juice ”.[1]
Smart people overreach too
Although admittedly ridiculous, this true story (with dialogue as I imagine it) has a lot to teach us about social science.[2]
Wheeler wasn’t acting on a whim. He had a theory: lemon juice works as invisible ink, so why not on faces? He had even tested it empirically, snapping a Polaroid of his juice-covered face and finding himself absent from the photo (he likely misused the camera). He was confident enough to bet his freedom on his theory. He was also catastrophically wrong.
It’s tempting to dismiss these robbers as exceptionally foolish. After all, this case inspired Dunning and Kruger’s research on overconfidence . But that misses the point. In a complex world, where countless forces interact, even the most rigorous of us can easily overestimate the reach of a given observation. When scholarship makes this error, it risks promoting “evidence-based” solutions that fail in practice.[3]
Can we identify such mistakes at scale? If they’re common, what can we do to correct them?
Narrative license – Stories can outrun evidence
Our team is obsessed with these questions . We believe social science can help society flourish, but we also believe that a mix of incentives and psychology ( publication pressures , funders rewarding impact , our taste for simple, coherent narratives ) make overreach challenging to avoid. The result is what we call “ narrative license ”: stories and practical advice that go beyond the evidence.[4][5][6][7][8][9]
Narrative license takes many forms. Articles can overgeneralise from a single measure to a broad concept, or from Western undergraduates to all of humanity. They can “ spin ” null results as positive. Or they can mistake correlation for causation.[10][11][12]
In our recent paper , we focus on that last one. Causal overclaiming is especially consequential because interventions depend on causal assumptions. If changing X supposedly improves Y, it’s critical to know whether the two are actually causally related or merely occur together.[13]
We screened 194K observational studies for causal language
Using large language models, we identified 194,631 social science articles with a “ cross-sectional ” design: observational studies that measure all variables at the same time, no repeated measures, randomisation, or quasi-experimental methods such as instrumental variables . These designs can reveal associations and control for other factors, but they offer little leverage for causal identification.[14][15]
We then developed an expert-validated classifier to check whether each article’s title or abstract, parts that receive disproportionate reader attention, made causal claims anyway.
Overclaiming is common, even in top journals, and rising over time
Nearly half (46.3%) of these observational studies made causal claims in their title or abstract, a strikingly high rate given their design.
Causal overclaiming over time. Source: Isch et al. (2026), Nature Human Behaviour, CC BY 4.0.
Perhaps more striking is the trend. After holding steady at around 20% of articles published from 1980 to 2000, the share tripled: over 60% of article abstracts from 2024 made causal claims.
Overclaiming isn’t confined to lesser outlets. Articles in journals with higher impact factors were more likely to use causal language, and this pattern was also evident across twenty elite journals spanning four fields. In contrast, the Journal of Quantitative Description, which explicitly discourages causal claims in descriptive work, showed less causal language (though still some).
Causal overclaiming (A) by journal impact factor, and (B) across 20 elite journals and JQD. Source: as above.
Wording shapes what readers believe
To find out whether this language matters for reader interpretation, we randomly assigned college-educated participants to read either an original abstract or one that was minimally rewritten to make only associational claims. Causal wording led readers to see stronger causal evidence. However, a simple fix worked just as well: an editorial note stating the study design and its limitations, inspired by prior work , reduced causal interpretations as much as the rewritten abstract. Even so, many readers inferred causal evidence across conditions, suggesting how strong causal intuitions are. There’s a lot more in the paper , which also examines full text and compares overclaiming in LLM-generated summaries with article abstracts.[16][13]
Our findings have important caveats
First, cross-sectional studies represent only one research design. Although 194K articles sounds like a lot, it’s a small fraction of all scholarship. Our findings do not show that overclaiming occurs across the social sciences writ large.
Second, dismissing correlational evidence carries its own risks. Ronald Fisher infamously doubted the link between smoking and lung cancer . As the quip goes: correlation is not causation, but it is correlated with causation. The challenge is balancing rigour against excessive skepticism.[17]
Social science must keep stories honest
Science requires some amount of narrative license. Human minds weren’t built to comprehend long spreadsheets; researcher interpretation is necessary to convert data into understanding. The question is how far those interpretations can travel before they outrun the evidence.
Our findings suggest that many scholars travel too far. We don’t see this as misconduct, but rather a human preference for simple stories that collides with incentives for impact.
Fortunately, practical solutions exist. Many journals already discourage causal overclaiming, but few specify which designs can support causal claims or which words imply causality. Editors could spell out both, for instance, that cross-sectional studies should avoid terms like “affects,” “leads to,” or “improves.” They can also attach brief methodological notes outlining each study’s design and limitations, which our experiment suggests can work. Automatic classifiers could flag overreaching claims at submission, prompting authors and reviewers to take a second look. Authors, for their part, can temper their language once they know to watch for this tendency.
The lesson of the lemon juice is not that we should stop drawing conclusions from evidence. It’s that evidence can feel far more decisive than it is. Responsible scholarship requires resisting that feeling so that the stories we tell other scholars, policymakers, and practitioners are as solid as they sound.
📑 This post draws on the author’s co-authored article, Quantifying the prevalence and impact of overreaching causal claims in social science , published in Nature Human Behaviour.[13]
This article gives the views of the author, not the position of LSE Impact or the London School of Economics. You are agreeing with our comment policy when you leave a comment.[18]