Speaker
Description
In preclinical animal studies, researchers often have a certain degree of freedom when it comes to selecting the exact statistical analysis strategy for their experiment. Ideally, this analysis strategy should be specified prior to the experiment (and preregistered, if possible), with sample size planning conducted in accordance with the chosen analytical approach. Sample size calculations performed for different potential analysis strategies may yield substantially different estimates of the required number of units to achieve a desired level of statistical power. In animal experiments in particular, achieving adequate statistical power with the smallest possible sample size is desirable for ethical, financial, and practical reasons. Consequently, when multiple analysis strategies are possible, researchers may calculate sample sizes for each strategy and select the one that requires the fewest animals to reach the desired statistical power. At first glance, such sample size minimisation appears both reasonable and ethically appealing.
However, sample size planning is often based on prior data (e.g., from pilot experiments or previously published studies), which may be affected by publication or follow-up bias to an unknown extent. As a result, sample size estimates derived from using these data are often too small to achieve the intended statistical power. Selecting the analysis strategy that yields the smallest of these underestimated sample sizes can further reduce the actual power of the study. This is problematic for reproducing preclinical animal trials because underpowered studies result in a higher number of false negatives, making it less likely that the results of a previously published study with a true effect can be reproduced. Minimising the sample size in this way may thus constitute a questionable research practice and could be more appropriately described as ‘sample size hacking’ (in analogy to ‘p-hacking’). In this project, we formalize and discuss this concept and conduct simulation studies to quantify the impact of ‘sample size hacking’ on statistical power across various scenarios.
96432302484