Learning objectives

This lecture will cover:

• Types of prediction
• 2x2 contingency table
• Metrics to evaluate model performance
• Model development and validation

in the context of binary risk prediction

Acknowledgements

Materials and data presented are based on previous slides prepared by L. Palla, D. Prieto, and E. Williamson

Scenario

Consider a dataset consisting some variables collected from a group of individuals. We want to predict which individuals develop a certain binary outcome $Y$.

Examples

*Note: prediction modelling is not limited to biomedical problems

Types of prediction: deterministic vs probabilistic

• Deterministic:

  • Classify each individual into one of the two possible outcomes.
  • Often used in (Supervised) Machine Learning

• Probabilistic:

  • Assign each individual a probability of developing the outcome.
  • Often used in Biostatistics and is known as Risk Prediction*

Both methods use individual-level data on a set of variables (or predictors) to develop a prediction model

Types of prediction: diagnostic vs prognostic

Example

Who will develop myocardial infarction in the next 5 years?

Deterministic classification

Suppose we have a pre-defined deterministic classification model C:

C(age, sex, diabet, SBP) = 0 or 1

Probabilistic prediction

Suppose we have a pre-defined probabilistic prediction model P:

P(age, sex, diabet, SBP) = [0, 1]

Observation

... after 5 years follow-up

Model validation

2x2 contingency table / Confusion matrix

2x2 contingency table / Confusion matrix

Sensitivity

• a.k.a True Positive Rate, Recall, Hit / Detection Rate

• Probability of correctly predicting positive outcome

• What is the probability of a positive classification, given a positive outcome?

• $P(\hat{Y}=1|Y=1$)

Specificity

• a.k.a Selectivity, True Negative Rate

• Probability of correctly predicting ngative outcome

• What is the probability of a negative classification, given a negative outcome?

• $P(\hat{Y}=0|Y=0$)

Positive predictive value (PPV)

• Probability of a positive outcome given a positive classification

• $P(Y=1|\hat{Y}=1$)

• For a given model (e.g. diagnostic test) with fixed sensitivity and specificity, PPV is positively correlated with disease prevalence

• Intuition: as prevalence increases, true positive increases while false positives decreases

Negative predictive value (NPV)

• Probability of a negative outcome given a negative classification

• $P(Y=0|\hat{Y}=0$)

• For a given model (e.g. diagnostic test) with fixed sensitivity and specificity, NPV is negatively correlated with disease prevalence

• Intuition: as prevalence increases, true negative decreases while false negative increases

Comparing classifications with observations

Suppose we have the following contingency table for classification model C:

Comparing classifications with observations

Suppose we have the following contingency table for classification model C:

Comparing predictions with observations

Cut-off point: Yes if P > 0.1

Higher sensitivity, lower specificity than classification model C

Cut-off point: Yes if P > 0.4

Cut-off point: Yes if P > 0.4

Same contingency table as classification model C

Cut-off point: Yes if P > 0.55

Cut-off point: Yes if P > 0.55

Lower sensitivity, higher specificity than classification model C

 All cut-off points

Receiver Operating Characterictic (ROC) Curve

A curve linking all the sensitivity against the specificity values

Area Under the [ROC] Curve (AUC / AUROC)

What is the AUC?

Area Under the [ROC] Curve (AUC / AUROC)

What is the AUC?

Sample calculation with yardstick package in R

# df = the dataset shown in previous slides
roc <- yardstick::roc_auc(df, truth = Observed, Predict)
roc

## # A tibble: 1 × 3
##   .metric .estimator .estimate
##   <chr>   <chr>          <dbl>
## 1 roc_auc binary         0.854

AUC = 0.8541667

AUC and the distributions of predictors in the two outcome groups at different cut-off values

by Dariya Sydykova (follow link for more info)

AUC as a measure of model discrimination

by Dariya Sydykova (follow link for more info)

What does AUC tell us?

• AUC estimates the probability that a randomly chosen observed “yes” was assigned a higher probability than a randomly observed “no” by the model

• Real world models will have AUC from 0.5 to 1. A value closer to 1 indicates better performance in separating "yes" and "no".

• For binary classification, AUC is equal to concordance (C) statistic

How do we come up with predictions?

• We propose a statistical model for the probability of the event happening $P(Y_i=1)$ depending on the other variables

• For example a logistic model:

$$\begin{array}{}\log \left(\frac{P(Y_i=1)}{1-P(Y_i=1)}\right)={\beta }_0+{\beta }_1{X}_i+{\beta }_2{Z}_i+\cdots & \text{(1)}\end{array}$$

• We need a training set where we can observe all the variables $Y_i,X_i,Z_i,\dots$ to estimate the coefficients ${\beta }_0,{\beta }_1,{\beta }_2,\dots$

• Once we have the coefficients that best fit the data we can calculate the predicted risk for each individual $i$

$$\begin{array}{}\hat{P}(Y_i=1)=\frac{e^{{\hat{\beta }}_0+{\hat{\beta }}_1{X}_i+{\hat{\beta }}_2{Z}_i+\cdots }}{1+e^{{\hat{\beta }}_0+{\hat{\beta }}_1{X}_i+{\hat{\beta }}_2{Z}_i+\cdots }}& \text{(2)}\end{array}$$

A larger example with 2000 individuals

We will use the variables Age, Sex, SBP, and BMI to predict if the person will be dead (Death = 1) or alive (Death= 0) in 5 years time

Model M1: Logistic regression

Stata command:

logistic dead age sex sbp bmi

Note that BMI is not statistically significant (P = 0.185)

Make predictions from Model M1

Create variable: logit of the probability of death - equation (1)

predict m1lp, xb

Create variable: predicted probability of death - equation (2)

predict m1pr

Predicted probability of death in dead and alive group

Model calibration

Evaluate goodness of fit with Hosmer-Lemeshow test

estat gof, group(10) table

Model calibration

Goodness of fit: Observed and expected events by deciles of risk

Limitation: Hosmer-Lemeshow test can not tell the direction of miscalibration and relies on arbitrary grouping

Model calibration with calibration plot

Contingency table: cut-off $P(Y=1)\ge 0.3$

estat classification, cutoff(0.3)

ROC curve from model M1

There is a 78% probability that a person who died is assigned a higher predicted risk by the model than a person was alive by the end of the follow up

Evaluating prediction model performance

Calibration

• The agreement between the predicted & observed outcomes
• For a group of patients with 10% predicted risk, do 10% experience the outcome?
• e.g. Goodness of fit test, calibration curve

Discrimination

• The ability of the model to distinguish between positive and negative outcome
• e.g. AUC / C statistic

Clinical usefulness

• Does the model provide accurate predictions at the patient level that can be used to guide clinical decision making?
• e.g. Decision curve analysis

Summary

• Prediction modelling can be broadly categorised into deterministic and probabilistic methods

• 2 x 2 contingency table / confusion matrix is useful as a first step to evaluate model performance

• AUC is a useful measure of the model discrimination

• Comparing observed and predicted risks is useful for model calibration

• Assessing clinical usefulness requires other approaches and often requires insights from beyond the data

References and further reading

• Textbook: Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer, New York, NY; 2009.

• Practical article: Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014 Aug 1;35(29):1925–1931.

• Reporting guide: Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration. Ann Intern Med. 2015;162:W1–W73.

• Comparison with machine learning: Breiman L. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Stat Sci. Institute of Mathematical Statistics; 2001 Aug;16(3):199–231.