Industrial Manual RAG Quality Prediction with Trace-Level Evidence Alignment on eManual

Abstract

Enterprise retrieval-augmented generation (RAG) systems require more than answer-level accuracy: operators need to know whether a response is supported by retrieved evidence, whether the retrieved context is useful, and whether the generator actually uses that context. This study examines industrial manual RAG quality prediction through trace-level evidence alignment, using the eManual subset of RAGBench as the primary customer-support setting and FinQA as a finance-domain transfer case. The analysis follows the TRACe formulation of adherence, relevance, utilization, and completeness and compares the published test-split outputs of a GPT-3.5 judge, RAGAS, TruLens, and a fine-tuned DeBERTa-v3 judge. Rankings, absolute gaps, relative changes, and all visual summaries are recomputed from the benchmark matrix. On eManual, DeBERTa-v3 reaches 0.76 hallucination AUROC, 0.15 above the strongest prompt-based baseline, and reduces relevance RMSE from 0.17 to 0.13. On FinQA, it reaches 0.81 hallucination AUROC, a 0.24 absolute gain, while utilization RMSE improves from 0.13 to 0.10. The comparison also reveals important exceptions: GPT-3.5 has lower utilization error on eManual, and RAGAS has the lowest relevance error on FinQA. These results support a multi-signal evaluation layer in which aggregate scores are linked to sentence-level evidence records. For industrial support systems, trace supervision provides the clearest benefit for unsupported-response detection and enables targeted remediation of retrieval, evidence use, and answer grounding.