Summary
Backbase's study demonstrates that a smaller, domain-specific AI model (12B parameters) can outperform a much larger general-purpose model (GPT-4.1) in live banking deployment. The key insight: training the model to refuse answers when evidence is insufficient (12% refusal rate) actually improved customer query resolution by 7.1 percentage points. The model was 20-50x cheaper to run (~$0.001/query) and cost only ~$1,800 to train. The research also found that data order matters more than data volume — training sequentially (general finance → calibrated refusal) produced 40%+ better results than combining all data at once. The work was led by Denys Katerenchuk (ex-Google, ex-IBM) through Backbase AI Research (acquired via Kasisto).
Key Points
- 12B parameter model vs GPT-4.1: +7.1pp customer query resolution (3,297 interactions, 7 months)
- Refusal rate: 12% (vs 4.3% untuned base, vs 20.2% GPT-4.1)
- Answer quality: 6.21/10 vs 5.72/10 for GPT-4.1
- Citation grounding: +2.3 points through source-document linking
- Cost: ~$0.001/query (20-50x cheaper than GPT-4.1); training: ~$1,800
- 22% of training examples had no correct answer — designed to teach refusal
- Data order > data volume: sequential training 40%+ better than combined training
- FinanceBench (SEC filings): stronger results than GPT-4.1
- Validated across 40+ financial institutions
- CEO Jouk Pleiter: "Saying 'I don't know' got treated as a weakness, not a feature"
- First published work from Backbase AI Research (Kasisto acquisition)