Before You Hit Submit: A Pre-Publication Reproducibility Audit Built From Researchers' Hard Lessons

The peer review notification arrived on a Tuesday afternoon. The reviewer had requested the raw data file and the analysis script. Standard enough. But when the researcher opened the folder where the files were stored, they discovered something that had gone unnoticed through months of drafting and revision: a step in the data cleaning process had been performed manually, outside the script, and never documented. The transformation was logical. The result was correct. But it could not be reproduced from the archived materials alone.

The paper survived peer review, but only after a difficult exchange and a substantially revised methods section. The experience, the researcher later said, was the most effective methodological training they had ever received—and the most expensive.

Stories like this one are more common than the published literature suggests, precisely because they rarely make it into print. Researchers who discover reproducibility problems before submission fix them quietly. Those who discover them afterward manage the consequences as discreetly as possible. The result is a kind of institutional silence around the specific, practical vulnerabilities that compromise reproducibility—a silence that leaves the next generation of researchers to learn the same lessons the hard way.

This article is an attempt to break that silence. What follows is a structured pre-submission audit—a checklist informed by the kinds of problems researchers have actually encountered, presented as a preventative tool that can be completed before a manuscript leaves your hands.

Section 1: Documentation and Transparency

The most common reproducibility failure is not analytical error. It is incomplete documentation. A study that was conducted correctly but documented incompletely is, from the perspective of an independent researcher attempting to replicate it, functionally indistinguishable from one that was conducted incorrectly.

Checkpoint 1: Can a researcher unfamiliar with your project reproduce your data pipeline from your archived materials alone? This is the foundational question of reproducibility. The honest answer requires more than reviewing your methods section—it requires sitting with your archived files and walking through the steps as if you had never seen them before. Better yet, ask a trusted colleague to attempt it.

Checkpoint 2: Are all data transformations, exclusions, and recoding decisions documented in your analysis script or a separate decision log? Manual data transformations performed outside a script are among the most common sources of undocumented decisions. Every exclusion criterion applied, every variable recoded, every outlier removed should appear explicitly in your documentation.

Checkpoint 3: Is your analysis script annotated sufficiently that its logic is clear to an external reader? A script that runs correctly is not the same as a script that is understandable. Annotations explaining why each step was taken—not just what the step does—dramatically improve the reproducibility of your analytic workflow.

Checkpoint 4: Are the software versions and package dependencies used in your analysis recorded? Analytic results can vary across software versions in ways that are subtle and difficult to diagnose after the fact. Recording the version of R, Python, SPSS, or any other software used—along with the versions of key packages—is a simple step that prevents significant confusion later.

Section 2: Sampling and Data Integrity

Checkpoint 5: Is your sampling procedure documented with sufficient detail that it could be replicated? This includes not only the eligibility criteria and recruitment strategy but also the order in which participants were approached, how refusals were handled, and how final sample composition was determined.

Checkpoint 6: Have you verified that your final analytic sample matches the sample described in your methods section? Sample sizes have a way of shifting between data collection and final analysis, and those shifts are not always reflected in the methods section. Compare the N in your analysis output against the N described in your manuscript before submission.

Checkpoint 7: If you collected data at multiple sites or time points, have you verified consistency in data collection procedures across those sites or points? Variation in how data was collected across sites or waves is a common but underappreciated threat to internal validity. A brief documentation review of site-level or wave-level procedures can surface inconsistencies that would otherwise appear only as unexplained variance in your results.

Section 3: Analytic Decisions and Pre-Registration

Checkpoint 8: Were your primary analyses specified before you examined the data? If your analysis plan was developed or substantially modified after data collection, your manuscript should clearly describe the exploratory nature of those analyses. If you pre-registered your study, verify that your reported analyses are consistent with your registration and that any deviations are explicitly noted.

Checkpoint 9: Have you examined whether your primary findings are robust to reasonable alternative analytic choices? Sensitivity analyses—running your primary analysis with alternative model specifications, different exclusion criteria, or alternative operationalizations of key variables—provide evidence that your findings are not an artifact of a single analytic decision. If they are not robust, that is important information.

Checkpoint 10: Are all reported statistics traceable to a specific line of code or calculation in your archived materials? Every number in your results section should have a clear origin. If a reviewer or replicator cannot locate the source of a reported statistic in your materials, it creates doubt about the reliability of your entire analysis.

Section 4: Measurement and Construct Validity

Checkpoint 11: Are the measures you used validated for the population you studied? Using an instrument validated in one population with a different population is a common and underappreciated validity threat. If your study population differs from the validation sample in ways that might affect how the construct is expressed or measured, that limitation should be explicitly acknowledged.

Checkpoint 12: If you developed or adapted measures for this study, is the development process documented? Custom or adapted instruments require documentation of the adaptation process, including any pilot testing conducted and the rationale for modifications made.

Section 5: Reporting Completeness

Checkpoint 13: Does your methods section describe your study as it was actually conducted, not as it was originally planned? Protocol deviations are common and not inherently problematic, but they must be reported. A methods section that describes the intended procedure rather than the executed procedure is a reproducibility problem waiting to be discovered.

Checkpoint 14: Have you reported all outcomes you measured, or only those that reached significance? Selective outcome reporting is one of the most consequential threats to the integrity of a scientific literature. If you measured outcomes that are not reported in your manuscript, the manuscript should acknowledge this and explain the basis for the selection.

Checkpoint 15: Is your data and code archived in a format and location that will remain accessible after publication? A link to a personal or institutional server that may not persist is not a reproducibility solution. Archiving materials in a stable repository—such as the Open Science Framework, Zenodo, or a discipline-specific archive—ensures that your materials remain accessible to researchers who encounter your work years from now.

Using This Checklist Effectively

This audit is most valuable when completed at least two weeks before your target submission date—early enough that any problems identified can be addressed without compromising your timeline. It is also most honest when completed by someone other than the primary analyst, whose familiarity with the data may cause them to overlook gaps that would be immediately apparent to an outside reader.

At Research Skill Center, we encourage researchers to treat pre-submission audits not as bureaucratic obligations but as a final opportunity to strengthen work they have invested significant effort in producing. The fifteen checkpoints above will not catch every possible vulnerability. But they will catch the ones that have most commonly surprised researchers who did not look for them—and that is a meaningful advantage to carry into any submission.