Trained to Fail: How Graduate Education Produces the Replication Crisis One Cohort at a Time

When a high-profile study fails to replicate, the public narrative tends to follow a predictable arc. A researcher is named, a journal issues a correction, and the broader scientific community reassures itself that the system of self-correction is working. What rarely gets examined is the question that sits beneath all of it: why did the original researcher not know their methods were fragile before the paper was ever published?

The uncomfortable answer, supported by a growing body of evidence in science education research, is that many of them were never taught otherwise.

The Myth of the Rogue Scientist

It is tempting to locate the replication crisis in individual moral failure. Fabrication and falsification do occur, and they deserve serious consequences. But large-scale replication efforts—including the Reproducibility Project in psychology and similar initiatives in cancer biology and economics—have consistently found that the majority of non-replicating findings cannot be attributed to deliberate misconduct. Instead, they trace back to underpowered study designs, analytic flexibility that was never disclosed, selective reporting that felt routine rather than deceptive, and statistical interpretations that were simply wrong.

These are not the signatures of bad scientists. They are the signatures of undertrained ones.

A 2019 survey published in PLOS ONE found that fewer than 40 percent of doctoral students in the social and behavioral sciences reported receiving formal instruction in statistical power analysis before completing their dissertations. A similar proportion said they had never been taught to pre-register a study or distinguish between confirmatory and exploratory analysis. These are not obscure technical skills. They are the foundational architecture of reproducible research, and they are being omitted from the training of an entire generation of researchers.

What Is Actually Missing

Doctoral programs in the United States vary enormously in structure and emphasis, but several methodological gaps appear with striking consistency across disciplines.

Power analysis and sample size justification. Many PhD students learn to run statistical tests without ever learning how to determine whether their study has adequate power to detect a real effect. The result is a literature saturated with underpowered studies whose positive findings are disproportionately likely to be false positives—a dynamic that statistician Andrew Gelman has described as the "winner's curse" of small-sample research.

Pre-registration and the confirmatory/exploratory distinction. The practice of specifying hypotheses, outcome measures, and analytic plans before data collection is one of the most effective tools available for limiting researcher degrees of freedom. Yet most graduate students encounter it only after joining a lab that has already adopted it, if at all. When analysis decisions are made after data are in hand, the line between hypothesis testing and hypothesis generating becomes invisible—and the resulting p-values mean something very different than they appear to.

Effect size interpretation and practical significance. Statistical significance and practical significance are not the same thing, yet graduate training often treats a p-value below 0.05 as the finish line rather than the starting point. Researchers who cannot interpret an effect size in context are poorly equipped to communicate the actual meaning of their findings—or to recognize when a result, while statistically significant, is scientifically trivial.

Documentation and data management. Reproducibility requires that another researcher could, in principle, follow your analytical path from raw data to published result. This demands systematic documentation of data cleaning decisions, variable transformations, and code. Most doctoral programs offer no formal instruction in research data management, treating it as a personal habit rather than a scientific obligation.

What Researchers Discover Too Late

Some of the most instructive perspectives on this problem come from researchers who have attempted to reproduce their own earlier work. The experience is more common than the literature suggests, and it is rarely comfortable.

One social psychologist, reflecting on a reanalysis of her dissertation data several years after publication, described realizing that two of her key analyses had been run multiple ways before she settled on the version that appeared in print—a process she had not documented and had not thought to disclose. "I wasn't trying to cheat," she noted. "I genuinely thought I was just being thorough. No one had ever told me that was the kind of decision that needed to be reported."

A neuroscience postdoctoral researcher described attempting to reproduce a key finding from his graduate work and discovering that the preprocessing pipeline he had used was never fully documented. He could approximate it but could not replicate it exactly—meaning that neither could anyone else. The original paper remained in the literature, unreproducible not by design but by omission.

These accounts are not exceptional. They are representative of a training environment in which methodological rigor is assumed to be transmitted through osmosis rather than explicit instruction.

A Curriculum Built for Prevention

The Research Skill Center's position is that methodological literacy is not an advanced topic to be introduced after researchers have already developed their professional habits. It is foundational, and it belongs at the beginning of doctoral training rather than as an afterthought.

A prevention-oriented curriculum would include several core components. First, every doctoral student should complete formal instruction in research design that explicitly covers statistical power, sample size justification, and the logic of hypothesis testing before they collect a single data point. Second, pre-registration should be introduced as a standard professional practice, not an optional reform. Programs should require students to pre-register at least one study during their training and to reflect on the difference between what they predicted and what they found.

Third, graduate programs should treat data documentation as a technical skill with explicit instruction, not an informal norm. This includes version control for analytic code, structured data management plans, and the archiving of materials sufficient for independent reproduction. Fourth, and perhaps most importantly, the culture of graduate training must shift away from treating replication failures as shameful anomalies and toward treating them as informative scientific events. Students who discover that their earlier work does not hold up under scrutiny should be supported in reporting that honestly—not incentivized to remain silent.

The Systemic Lever

The replication crisis will not be resolved by identifying and punishing researchers who cut methodological corners. Most of them were never given the tools to recognize the corners in the first place. The systemic lever is training—specifically, the willingness of doctoral programs, funding agencies, and professional associations to treat methodological education as a core scientific competency rather than a peripheral concern.

The researchers currently in graduate programs will produce the literature of the next two decades. What they are taught now about rigor, transparency, and reproducibility will shape the credibility of science itself. That is not a problem for journals and oversight committees to manage after the fact. It is a curriculum design problem, and it is solvable.