Speaker
Description
Statistical prediction models for binary outcomes are becoming increasingly popular. One signifi‐
cant challenge is calibrating these models to suit the characteristics of a target population that is
structurally different from the original population. Calibration is especially challenging when there
is no training data available from the target population. To address this problem, we propose a novel
calibration method, SimCal, which uses synthetic data generated from the model development data
in conjunction with marginal statistics from the calibration cohort. We show that expert‐judgment
modeling (EJM) may be used for calibration if cross‐sectional data from the target population are
available comprising expert judgments about the potential outcome and the covariates. We de‐
scribe three alternative calibration approaches when calibration data are lacking: similarity‐binning
averaging (SBA), adaptive calibration of predictions (ACP), and Elkan calibration. In a simulation
study, we compare SBA, ACP, Elkan calibration, and SimCal. R code for applying these methods
is provided from the re‐analysis of data on coronary artery disease. We illustrate all 5 calibration
approaches with a real data set for predicting functional outcome after stroke. None of the ap‐
proaches performed convincingly in all situations. SimCal performs well when model parameters
are correctly specified. EJM failed on the stroke data. Further research is urgently required for cali‐
bration in the absence of calibration data.
75002900909