Summary
Backbase published a study showing its 12-billion-parameter banking AI model outperformed GPT-4.1 in a live deployment at a large US financial institution. Over seven months, the model resolved 7.1 percentage points more customer queries across 3,297 interactions. The model was trained to refuse answers when evidence did not support a response (12% refusal rate vs 4.3% untuned vs 20.2% GPT-4.1). It scored 6.21 vs 5.72 on answer quality (10-point scale), with citation grounding improved by 2.3 points. Operating cost: ~$0.001/query (20-50x cheaper than GPT-4.1), training cost: ~$1,800. The research was led by Denys Katerenchuk (ex-Google, ex-IBM).
Key Facts
- 12B parameter model vs GPT-4.1; resolved 7.1pp more customer queries
- 3,297 interactions over 7 months at a large US financial institution
- Refusal rate: 12% (vs 4.3% untuned base, vs 20.2% GPT-4.1)
- Answer quality: 6.21/10 vs 5.72/10 for GPT-4.1
- Citation grounding: +2.3 points through source-document linking
- Cost: ~$0.001/query (20-50x cheaper than GPT-4.1); training: ~$1,800
- 22% of training examples had no correct answer — designed to teach refusal
- Data order mattered more than data volume: sequential training (general finance → calibrated refusal) beat combined training by 40%+ in answer quality
- FinanceBench (SEC filing questions): stronger results than GPT-4.1
- Backbase AI Research team joined through Kasisto acquisition
- CEO Jouk Pleiter: "Saying 'I don't know' got treated as a weakness, not a feature"
Why It Matters
Backbase's study challenges the assumption that bigger general-purpose models are always better for banking. The key insight: a smaller, domain-specific model trained to know its limits (higher refusal rate) can outperform a much larger general model on actual customer query resolution. The cost advantage (20-50x cheaper) makes this economically significant for banks processing millions of queries. The finding that data order matters more than data volume is a practical lesson for any financial institution building custom AI. This could accelerate the trend toward specialized banking AI models rather than relying on general-purpose LLMs.