LLM-Assisted Causal Attribution of Service Performance Upgrades on Churn and Tenure: Full Evaluation on the IBM Telco Customer Churn Dataset

Abstract

Service performance upgrades are frequently deployed to reduce customer churn, yet their real impact is hard to attribute from observational data because product selection, pricing, and customer preferences act as confounders. This paper proposes an LLM-assisted causal attribution workflow that converts a business-change description into an explicit causal question, a directed acyclic graph (DAG) with variable roles (confounder/mediator/outcome), and an auditable adjustment strategy. We then execute a full empirical evaluation on the IBM Telco Customer Churn dataset (7,043 customers), focusing on broadband customers (n=5,517) and operationalizing a ‘performance upgrade’ as adoption of TechSupport (T=1) versus no TechSupport (T=0). Outcomes are (i) churn within the last month and (ii) tenure in months. Under the backdoor criterion and positivity, we estimate average treatment effects (ATE) and heterogeneous treatment effects (HTE) using propensity score matching (PSM), inverse probability weighting (IPW), doubly robust augmented IPW (AIPW), double machine learning (DML), and a causal-forest-style DR-learner. Across estimators, TechSupport reduces churn by 7.3–10.0 percentage points and increases tenure by 2.4–4.4 months after adjustment; the cross-fitted AIPW estimate is +4.40 months (95% CI [3.39, 5.42]) for tenure and −0.092 (95% CI [−0.119, −0.065]) for churn risk. HTE analysis shows the largest benefits for month-to-month contracts (churn −0.147) and senior citizens (churn −0.156). Overlap/balance diagnostics confirm adequate propensity overlap and strong post-adjustment balance (max |SMD| reduced from 0.856 to ≤0.103). Sensitivity analysis (Oster δ) quantifies the level of unobserved selection required to explain away the estimated effects.