Speaker
Description
For a given research question and observational dataset, there are often numerous ways to specify the data analysis pipeline that leads from raw data to the result of interest. Data analysts must make a series of choices concerning data preprocessing, variable definitions, and statistical model specifications. For example, analysis pipelines may differ in their inclusion or exclusion criteria (e.g., the exclusion of a small subgroup of patients suspected to behave differently), in preprocessing steps such as data transformation (e.g., log-transformation or collapsing categories of categorical variables), or in the methods used for imputing missing values. They may also vary in the selection of adjustment variables in a multivariable regression model when estimating an effect of interest. In this work, we use the term "researchers’ degrees of freedom" to denote these analytic choices required when specifying a complete data analysis pipeline, and focus on studies that involve hypothesis testing and effect estimation for explanatory purposes.
We demonstrate how a class of methods known as multiple marginal models (MMM) - originally developed to control for multiple testing in the context of different-scaled, multiple correlated endpoints - can be adapted to address multiplicity arising from the researchers’ degrees of freedom described above. Specifically, we propose that researchers may explore various analytical specifications and focus on the one yielding the smallest p-value, provided they appropriately adjust for the resulting multiplicity of tests using the MMM framework. This approach allows analytical flexibility and adaptation to the data at hand (as opposed to strict statistical analysis protocols specified in advance), while preserving the nominal Type I error rate (as opposed to the practice known as "p-hacking") and provides a single interpretable result (as opposed to the reporting of the results of a multitude of analysis pipelines). It is illustrated through real data examples for various types of degrees of freedom, including but not limited to tuning parameters of statistical methods, the handling of outlying values, missing values and transformations of variables, and the specification of adjustment covariates in multivariable regression.
21429407209