Sooahn Shin (Applied Stats Workshop)

Speaker & Title

 Sooahn Shin (Harvard),  "Triage Score: A Counterfactual Risk Assessment Instrument"

Abstract

Risk assessment instruments, or "risk scores," are widely used in high-stakes decision-making settings such as medicine and the criminal justice system. A risk score predicts the likelihood of an undesired outcome if no intervention is made. Thus, a sufficiently high score is often interpreted as a recommendation to intervene. However, risk scores fail to account for what would happen if a decision-maker does intervene. This is problematic because effective decision-making requires consideration of how intervention affects outcomes. We propose "triage scores," which are based on counterfactual utilities. Unlike risk scores, triage scores incorporate counterfactual outcomes under alternative decisions, enabling decision-makers to incorporate a wide range of ethical and practical factors. We illustrate the use of triage scores with an application to our own randomized controlled trial evaluating a pre-trial risk assessment instrument. Our analysis demonstrates that triage scores are able to capture richer utility structures than risk scores and yield substantively distinct results regarding policy evaluation and learning.