Naoki Egami (Applied Stats Workshop)

Speaker & Title

Naoki Egami, "Conformal Policy Learning with Distribution-Free Safety Guarantees: Application to AI-Powered Interventions"

Abstract

Generative AI is emerging as a new class of intervention in the social sciences, with applications designed to change attitudes and behaviors through scalable, personalized interactions. For example, conversational agents have been used to reduce political polarization and improve workplace productivity. At the same time, recent empirical studies highlight an important risk: while such interventions may benefit many individuals and tasks, they may also harm others. How, then, can AI interventions be deployed safely? In this paper, we develop a new statistical framework, conformal policy learning, to deliver pre-specified safety guarantees when deciding whether individuals should receive a new intervention or the status quo. For instance, a researcher may require that the probability that an individual is harmed by the chosen intervention is below 1%. Using tailored conformal hypothesis testing, our method provides finite-sample safety guarantees under the standard exchangeability assumption, without relying on any modeling assumptions. It also achieves asymptotically optimal power or welfare maximization when the conditional expectation functions of outcomes are correctly specified. Thus, our treatment assignment rule is guaranteed to be safe in finite samples while attaining optimality under standard modeling assumptions. In practice, our framework enables researchers to deploy AI safely by assigning AI interventions only to people and tasks that satisfy user-specified safety requirements, and by reverting to the status quo otherwise. This offers a middle ground between two undesirable extremes: unfiltered deployment that ignores AI risks and total avoidance due to safety concerns. We illustrate the method through extensive simulations and an experiment in which randomly assigned AI chatbots are used to reduce conspiracy beliefs.