Uncertainty-Aware Uplift Modeling for Safer Marketing Targeting: Conformal Prediction and Bayesian Calibration with LCB Policies

Abstract

Uplift models estimate the incremental impact of an intervention (e.g., a marketing message) on an outcome, and are widely used to allocate limited treatment budget. In practice, incremental effects are small and noisy, so point-estimate uplift targeting can be unstable: aggressive targeting may “flip” ROI from positive to negative when the true uplift is near zero. This paper studies uncertainty-aware uplift targeting for operational risk control. We convert point uplift estimates into calibrated uncertainty intervals using two complementary approaches: (i) normalized split conformal prediction on transformed-outcome regression, yielding distribution-free marginal coverage for pseudo-outcomes; and (ii) Bayesian binned calibration, yielding interpretable credible intervals for bin-level uplift via Beta posteriors over treated/control conversion rates. We then deploy a conservative lower confidence bound (LCB) policy that ranks or filters customers by LCB uplift to trade expected gain for reduced downside risk. We conduct full empirical evaluations on the Kevin Hillstrom MineThatData e-mail campaign dataset (64,000 customers) under two binary-treatment tasks: Mens E-Mail vs No E-Mail and Womens E-Mail vs No E-Mail, with conversion as the primary business outcome. For 90% nominal intervals, normalized conformal achieved 0.904 and 0.899 marginal coverage on the transformed outcome for the Mens and Womens tasks, respectively, while producing heterogeneous interval widths that reflect feature-dependent noise. In policy evaluation via inverse propensity scoring, LCB-conformal targeting improved the best 95% lower bound profit from 15.1 to 21.9 per 10,000 customers on the Mens task compared with point-estimate targeting at the same cost assumptions. On the Womens task, point-estimate targeting delivered the highest risk-adjusted profit, while LCB policies were more conservative and reduced expected profit. Overall, uncertainty estimates are essential for explaining and controlling risk; when interval width correlates with ranking mistakes, LCB targeting materially reduces “bad surprise” outcomes.