OpsLLM for Cloud Incident Triage: Bilingual RAG-Based Root Cause Analysis and Alert Summarization for AI Infrastructure Operations

Abstract

Cloud incidents in AI infrastructure involve overloaded accelerators, brittle service dependencies, distributed databases, logs, and networking paths. Operators need answers that are fast, grounded, bilingual, and auditable. This paper presents OpsLLM, a bilingual retrieval-augmented triage pipeline for root-cause analysis and alert summarization. The study evaluates the pipeline on a normalized OpsEval-format corpus containing 7,184 multiple-choice root-cause records and 1,736 question-answering alert-summary records. The corpus is split into 1,798 MCQ development records, 5,386 MCQ test records, 439 QA development records, and 1,297 QA test records, and all reported values are produced by the included reproducibility script with seed 20260510. Six methods are compared: zero-shot constrained decoding, few-shot exemplar voting, BM25 RAG, dense retrieval RAG, hybrid RAG, and self-consistency voting. The dense RAG path achieved 96.19% top-1 MCQ accuracy, hybrid RAG achieved 95.79%, and self-consistency achieved 96.10%; zero-shot decoding reached 76.29%. On the QA alert-summary task, hybrid RAG achieved the strongest root-cause exact score at 85.81% and the best ROUGE-L at 79.41%. The results show that bilingual normalization and retrieval are decisive for operational triage, while self-consistency improves robustness only when retrieval paths provide complementary evidence rather than correlated evidence. The paper includes detailed metrics, retrieval curves, latency and token-cost estimates, domain-level analysis, residual error analysis, and executable code for reproducing all tables and figures.