Comparing Prompting Strategies and Backbone Choices for LLM Agents on Tabular Credit-Card Fraud Detection: An Empirical Study

Abstract

Large language model (LLM) agents are increasingly proposed for financial decision tasks, yet their behaviour on tabular fraud detection has not been measured under a controlled protocol. This paper reports a head-to-head empirical comparison of four prompting strategies (zero-shot, few-shot, chain-of-thought, and ReAct-style tool use) across four LLM backbones (GPT-4o-mini, Claude-3.5-Haiku, Llama-3.1-8B-Instruct, and Qwen-2.5-7B-Instruct) on two public datasets: the ULB credit-card corpus with anonymised principal-component features and the Bank Account Fraud (BAF-Base) suite with human-readable attributes. All 32 agent configurations are benchmarked against logistic regression, random forest, XGBoost, and LightGBM under identical temporal splits. Gradient-boosted trees retain a clear advantage: XGBoost reaches an AUC-PR of 0.847 on ULB and 0.196 on BAF, while the strongest agent configuration attains 0.412 and 0.153. The accuracy gap narrows substantially when features carry semantic meaning, and tool-augmented prompting yields its largest gain (+0.040 AUC-PR over chain-of-thought) on anonymised features. Inference cost remains three to four orders of magnitude higher for agents. A taxonomy of 400 manually coded errors indicates that agents most often miss fraud expressed through combinations of weak signals. Practical guidance for hybrid deployment is derived from these measurements.