1
posts
2
3
4
5
6
7
8
9
10
11
12
Quantifying the prevalence and impact of overreaching causal claims in social science
(www.nature.com)
In this study, we have demonstrated that overreaching causal language is pervasive across the social sciences. Restricting our analysis to non-experimental studies that rely exclusively on cross-sectional data, we find that nearly half (46.3%) of such articles published since 1980 contained causal claims in either their titles or abstracts. Even this large number understates the prevalence in recent years, which is over 60%, reflecting a threefold increase since 2000—a trend that occurred across all disciplines, albeit to different extents. Interestingly, these results applied equally to the most prestigious and selective journals and somewhat more so to high-impact journals in general. Moreover, full-text analyses showed that even when abstracts used cautious, associational language, corresponding articles often included direct causal statements elsewhere. Together, these results, which align with other evidence of causal overclaiming in psychology51 and biomedical science35, indicate that rhetorical overreach is not limited to isolated cases or to low-status outlets but reflects a systemic feature of how correlational findings are communicated.
In addition to studying the prevalence of inappropriate causal claims, we also studied their effects on humans and language model summaries in a series of three experiments in which we also tested various interventions to improve reader inferences. For humans, we found that our interventions somewhat reduced the share of respondents who agreed that the paper in question provided causal evidence but that respondents continued to perceive causality at surprisingly high rates in all conditions. Even when abstracts were rewritten in strictly associational terms or corrective notes were added to the text, participants often reintroduced causal phrasing in their summaries. These findings suggest that causal interpretation is deeply ingrained in humans and is difficult to eliminate even with more careful phrasing. Our experiments revealed that LLMs frequently introduced causal language when summarizing study findings, under basic, simplifying and practical-oriented prompting strategies. This mirrors recent evidence that LLMs overclaim when summarizing scientific findings, for instance, by overgeneralizing52.
...
Fig. 2: The annual proportion of cross-sectional articles that contain causal language across all articles (black) and by discipline (colours).
Overall, the proportion remained stable at roughly 20% from 1980 to 2000, then increased rapidly tripling by the 2020s. This increase appears across all disciplines, with notable recent variation: business shows the highest rates, whereas sociology shows the lowest. The grey band represents the annual 95% Wald CI for all articles. The horizontal dashed line marks the pooled overall proportion across all articles (46.3%, N = 194, 631), which is disproportionately influenced by the higher volume of articles in later years.
13
14
15
16
17
18
Loneliness Rewires the Brain to Drive Alcohol Consumption, Mouse Study Finds
(news.feinberg.northwestern.edu)
19
20
21
22
23
24
25
view more:
next ›