Real-Time AI Risk Scoring

The Challenge

A Series B fintech company was losing customers at an alarming rate — not to competitors, but to their own fraud detection system. Their rules-based model was blocking 23% of legitimate transactions, generating thousands of false positives daily. Customer service was overwhelmed with disputes. Churn was accelerating. The business was at risk.

The existing system was a patchwork of 400+ hand-crafted rules accumulated over five years. Adding new rules to reduce false positives inevitably increased false negatives (actual fraud). The system had become unmaintainable, and the team had lost confidence in their ability to improve it without breaking something else.

The Solution

We replaced the rules engine with a two-stage ML system: a fast gradient boosting model for initial scoring (sub-10ms), followed by a slower but more accurate neural network for borderline cases. The system was designed to be explainable — every decision came with a ranked list of contributing factors that customer service could use to resolve disputes.

The training pipeline used three years of historical transaction data, with careful attention to class imbalance and temporal leakage. We implemented a shadow mode deployment that ran the new system in parallel with the old one for 30 days before cutover, allowing us to validate performance on real traffic without risk.

The infrastructure was built on a streaming architecture using Kafka and Redis, with the ML models served via a custom inference layer optimized for the latency requirements. We designed the system to degrade gracefully — if the ML layer was unavailable, it fell back to a simplified rules engine rather than blocking all transactions.

Tech Stack