Page 322 - Contributed Paper Session (CPS) - Volume 2

CPS1874 Yiyao Chen et al.
depended on heterogeneity of the data sets. These methods remain largely
descriptive.
We propose a more formal method for quantifying validation that borrows
philosophy from casual inference theory and principal stratum analysis for
analyzing effects of treatments based on observational studies (Rubin, 1974;
Frangakis & Rubin, 2002). Our method requires training and test sets that
include both patients that did undergo the diagnostic procedure for outcome
determination as well as the patients that did not, the latter of which are
typically not included in the training nor test set. The method assumes a risk
model built on the patients that underwent the diagnostic procedure in the
training set, with the risk factors required for the model available for all
patients, in both the training and test sets, ascertained or not. Conditional on
observed covariates we define the principal stratum of patients who would
have undergone disease ascertainment in either the training or test set, and
propose operating characteristics of the risk model be evaluated on this
stratum in addition to just the test set of ascertained patients.
We illustrate the method with a risk model for predicting prostate cancer
on biopsy developed on the Prostate, Lung, Colorectal, and Ovarian Trial
(PLCO, Andriole et al. (2009)) and tested on the Selenium and Vitamin E Cancer
Prevention Trial (SELECT, Klein et al. (2011)).

2. Methods
Suppose there are a total of N participants, assumed to be independent,
for which some will be used in a training set to develop a prediction model
and the remaining to test the model. For in {1,2, … , }, let be the indicator

equal to one for patients belonging to the test set and 0 for the training set,
and the vector of risk factors assumed to be the fixed for participants

irrespective of whether they were in the training or test set. Therefore, for any
risk model R depending on only these risk factors, the individual patient risk
( ) is identical regardless of whether the patient falls in the test or training

set. We will here presume the risk model R was built on the training set, using
established risk factors that would be ordinarily assessed in any cohort, and
thus monotonic functions of the fixed risk factors X observed in the cohorts.
Let ( ) be the biopsy indicator equal to 1 if the participant had a biopsy

performed and 0 otherwise, and ( ) the indicator equal to 1 if the patient

has prostate cancer and 0 otherwise. Note that ( ) is only observed for

( ) = 1 and we assume no measurement error in the biopsy.

We translate the first assumption typically used in causal inference, Stable-
Unit-Treatment-Value Assumption, in this case that the biopsy and cancer
outcomes of a participant are not affected by the training versus test set
assignment of the other participants (Rubin, 1980).
311 | I S I W S C 2 0 1 9