Page 321 - Contributed Paper Session (CPS) - Volume 2

CPS1874 Yiyao Chen et al.

A causal inference approach for model validation
1,2
1
Yiyao Chen , Donna P. Ankerst
1 Department of Life Sciences, Technical University of Munich, Freising, Germany
2 Department of Mathematics, Technical University of Munich, Garching b. Munich, Germany

Abstract
External validation of clinical risk models on independent populations is a
necessary procedure for ensuring such models are ready for public use.
Limited procedures exist for quantifying the difference between training data
sets used to develop risk models and test sets to validate them, as well as how
to translate these differences to explain differences in operating characteristics
across test sets. We propose a novel validation method based on causal
inference theory that identifies a principal stratum of patients that would have
had their disease status ascertained in both the training and test sets. We
propose evaluation of the risk model on this stratum to provide an
unblemished evaluation of the reproducibility of the model that can be
compared with the observed operating characteristics on the test set. We
illustrate the method with a risk model for predicting prostate cancer by
biopsy developed on the Prostate, Lung, Colorectal, and Ovarian Trial and
tested on the Selenium and Vitamin E Cancer Prevention Trial.

Keywords
potential outcome framework; principal stratum; true positive rate

1. Introduction
External validation results of a risk model developed on a training set and
evaluated on a test set are a mixture of the quality of the risk model as well as
the homogeneity between the training and test sets, the latter often referred
to as the case-mix difference. In construction of a contemporary online
prostate cancer risk calculator based on multiple international heterogeneous
cohorts, Ankerst et al. (2018) described the divergence between cohorts in
terms of statistical tests of differences in risk factors distributions as well as
graphical displays of odds ratios of outcome prevalence by risk factor
prevalence across cohorts. Debray et al. (2015) applied multivariate logistic
regression with cohort assignment as the response to detect heterogeneity in
risk factor distribution between training and test cohorts to better understand
the validation results. van Klaveren et al. (2016) introduced a model-based
c−statistic that did not rely on observed outcomes from the test set, with the
motivation that the change in values from the training to test set only

310 | I S I W S C 2 0 1 9