Speaker
Description
Low rates of replicability in early phase biomedical research hinder progress and putatively cause high attrition rates in clinical trials. To improve evidence generation processes, preclinical confirmatory studies and preregistration offer potentially effective strategies. By comparing conduct and outcome of preclinical studies utilizing such strategies, we examined how different degrees of rigor improve evidence generation.
We evaluated experimental rigor of preclinical studies by extracting measures to reduce risk of bias, sample sizes, and effect sizes of primary outcomes of three different preclinical data sets. A. Preregistered single-laboratory animal studies of two study registries. B. Published multi-laboratory animal studies, extending an existing dataset. C. A unique set of confirmatory multi-lab projects including associated single-lab exploratory data from a dedicated German funding call.
Methodological rigor in reported results increased across all three data sets relative to exploratory experiments. In preregistered studies, time from preregistration to publication was faster in experiments with high methodological rigour. Reliability qua sample size of preregistered experiments was heterogenous with frequently optimistic effect size assumptions. In confirmatory experiments, sample sizes were increased in comparison to the exploratory phase resulting in higher reliability. Effect sizes were decreased in multi-laboratory and confirmatory studies. In preregistered studies effect sizes were also lower than preregistered values. The magnitude of this decrease was positively associated with measures of experimental rigour. In the confirmatory studies, approximately 80% of experiments failed to generate hypothesis confirming evidence on several replication assessment criteria. Similar results were obtained in multi laboratory studies and preregistrations. A follow-up in-silico investigation into the test severity of diverse replication criteria revealed large differences in sensitivity and specificity between criteria.
Our preregistered side-by-side analysis of three complementary data sets that employ rigorous research practices such as preregistration and multi-laboratory study design demonstrates how such strategies meaningfully contribute to preclinical evidence generation processes. Decisions to move to clinical trials will benefit from particularly confirmatory multi laboratory designs. Such trials provide relevant information on efficacy and are well suited to effectively reduce decision uncertainty.
53573501267