Statistical Power, Effect Sizes, and Sample Size: The Planning Decisions That Determine Whether Your Research Can Find What It's Looking For

Imagine spending six months designing a survey, recruiting participants, collecting responses, and running your analysis—only to find that your study was never statistically capable of detecting a real effect, even if one existed. This outcome, more common than most introductory methods courses acknowledge, is the direct result of underpowered research. It produces inconclusive findings, wastes resources, and contributes to a literature cluttered with null results that may reflect design limitations rather than true absences of effect.

Understanding statistical power, effect sizes, and sample size is not optional knowledge for researchers who want their work to be credible and publishable. These concepts are interconnected, and they must be addressed during the planning phase—not after data collection, when it is too late to do anything about them.

What Statistical Power Actually Means

Statistical power is the probability that your study will detect a true effect when one genuinely exists. Formally, it is expressed as 1 minus beta (1 − β), where beta represents the probability of a Type II error—failing to reject a false null hypothesis, or in plain terms, missing a real effect.

The conventional benchmark for acceptable power is 0.80, meaning an 80% probability of detecting a true effect. This threshold, established largely through the influence of statistician Jacob Cohen, is now a standard expectation in most peer-reviewed journals and funding agencies, including those affiliated with the National Institutes of Health (NIH) and the National Science Foundation (NSF). A study with power below 0.80 is considered underpowered and will face scrutiny during peer review.

Four factors determine statistical power, and they are mathematically linked:

1. Alpha level (α): The significance threshold you set, typically 0.05
2. Effect size: The magnitude of the relationship or difference you expect to find
3. Sample size: The number of observations in your study
4. Power (1 − β): The sensitivity of your design

When you fix three of these values, the fourth is determined. This relationship is the foundation of power analysis.

Understanding Effect Size Before You Choose a Sample

Effect size is a measure of the practical magnitude of a relationship or difference, independent of sample size. This distinction matters enormously. A study with a very large sample can detect a statistically significant effect that is substantively trivial—so small that it has no meaningful real-world implication. Conversely, a genuinely important effect may go undetected in an underpowered study simply because the sample was too small to resolve it.

The most commonly used effect size metrics depend on the type of analysis:

• Cohen's d is used for comparing two means. A d of 0.2 is considered small, 0.5 medium, and 0.8 large.
• Pearson's r quantifies the strength of a correlation. Values of 0.1, 0.3, and 0.5 correspond to small, medium, and large effects, respectively.
• Eta-squared (η²) and partial eta-squared are used in ANOVA contexts. Values of 0.01, 0.06, and 0.14 are the small, medium, and large benchmarks.
• Odds ratios are standard in logistic regression and epidemiological research.

When planning a study, you need to specify an expected effect size before calculating the required sample. There are two ways to arrive at this estimate: draw from prior literature in your area (meta-analyses are particularly useful here), or conduct a sensitivity analysis to determine the minimum effect size your study could detect at a given power level and sample size.

Relying on Cohen's generic benchmarks alone—without grounding them in the specific literature of your field—is a common shortcut that can lead to miscalibrated designs. A medium effect in educational psychology may be quite different from a medium effect in pharmacology.

How to Conduct a Power Analysis

A power analysis is a calculation performed before data collection that determines the sample size needed to achieve your target power given a specified effect size and alpha level. It can also work in reverse: given a fixed sample size, it tells you the minimum detectable effect size or the power your study will achieve.

For researchers in the United States, several free tools make this calculation accessible:

• G*Power (available at gpower.hhu.de) is the most widely used standalone software for power analysis across a broad range of statistical tests. It handles t-tests, ANOVAs, correlations, regression, and more.
• R packages such as pwr, WebPower, and simr (for mixed-effects models) provide flexible, scriptable alternatives for more complex designs.
• PASS (Power Analysis and Sample Size) is a commercial option frequently used in clinical and biomedical research settings.

To walk through a basic example: suppose you are planning a two-group comparison (independent samples t-test) and you expect a medium effect size (d = 0.5) based on prior studies. You want power of 0.80 and an alpha of 0.05. Running this in G*Power yields a required sample of approximately 64 participants per group—128 total. If your recruitment capacity is limited to 80 participants total, your power drops to roughly 0.56, meaning you will miss a real effect nearly half the time. That is a study worth redesigning before it begins.

Sample Size Is Not Just About Power

While power analysis is the primary driver of sample size decisions, it is not the only consideration. Several additional factors should inform your planning:

Attrition and missing data. In longitudinal studies or intervention trials, participant dropout is predictable. Your target sample size should be inflated to account for anticipated attrition—typically by 10–20%, depending on the study context.

Subgroup analyses. If your research questions involve comparisons across demographic groups (by race, gender, geographic region, etc.), each subgroup must be adequately powered independently. A study powered for the full sample may be severely underpowered for any single subgroup.

Multiple comparisons. When testing multiple hypotheses simultaneously, the probability of at least one false positive increases. Corrections such as Bonferroni adjustment or false discovery rate (FDR) control effectively raise the bar for significance, which in turn requires larger samples to maintain power.

Measurement reliability. Instruments with lower reliability (lower Cronbach's alpha, for example) introduce more measurement error, which attenuates observed effect sizes. If your measures are imperfect—and they usually are—you may need a larger sample than the true effect size alone would suggest.

Integrating Power Planning Into Your Research Workflow

Power analysis should not be an afterthought added to a grant application or methods section after the study is already designed. It is most valuable as a constraint that shapes the entire architecture of a project.

A practical workflow looks like this:

1. Define your primary research question and the statistical test it requires.
2. Identify a defensible expected effect size from prior literature or a meta-analysis.
3. Set your alpha level (typically 0.05) and target power (typically 0.80, or 0.90 for high-stakes research).
4. Run a power analysis to determine the required sample size.
5. Assess feasibility: can you recruit that sample within your timeline and budget?
6. If not, revise the scope of the study, seek additional resources, or conduct a sensitivity analysis to document the minimum detectable effect given your constraints.
7. Document the entire process in your pre-registration or methods section.

Funding agencies and journal editors increasingly expect to see this documentation. The NIH, for instance, requires justification of sample size in grant applications as part of its rigor and reproducibility standards. Demonstrating that your study is appropriately powered is no longer a bonus—it is a baseline expectation.

The Cost of Skipping This Step

Underpowered research does not simply produce inconclusive results. It distorts the scientific record. When low-powered studies do detect significant effects, those effects are statistically more likely to be overestimates—a phenomenon sometimes called the winner's curse. This contributes to the replication problems that have troubled fields ranging from social psychology to nutrition science.

Conversely, a well-powered study that finds a null result is genuinely informative. It tells the field something meaningful: the effect, if it exists, is smaller than the study was designed to detect. That is useful knowledge. An underpowered null result tells you almost nothing.

Investing time in power analysis before data collection is one of the highest-return activities available to a researcher at the planning stage. It is not glamorous work, and it rarely makes it into an acknowledgments section. But it is the difference between a study that can answer its own question and one that, despite all the effort invested, cannot.