Checkr: Streamlining Background Check Classification with Fine-tuned Small Language Models

Overview

Checkr automated adjudication of background checks using a tuned logistic regression model for 98% of cases, but the remaining 2% required classifying noisy records into 230 categories. After testing GPT-4 (80-82% accuracy on hard cases, ~15s latency, ~$12k cost) and GPT-4 with RAG, Checkr fine-tuned Llama-2-7b (85% accuracy) and then, in production on Predibase, Llama-3-8b-instruct, achieving 90% accuracy on the hardest 2% of cases with 0.15-second response times (30x faster than GPT-4) and a 5x cost reduction, using LoRA/LoRAX for parameter-efficient fine-tuning and multi-adapter serving.

The challenge

Checkr's automated adjudication used a tuned logistic regression model to reduce manual reviews by 95% for 98% of its data, but the remaining 2% of cases involved noisy, unstructured text data requiring classification into 230 distinct categories; their original Deep Neural Network solution could classify only 1% of these cases with decent accuracy, leaving the rest requiring customer intervention, all while needing to meet low-latency SLAs and control inference costs across millions of tokens processed monthly.

The solution

Checkr first tested GPT-4 as a general-purpose Expert LLM (80-82% accuracy on the hardest 2% of cases, ~15s latency, ~$12k cost) and GPT-4 with RAG (accuracy decreased on the hardest cases). They then fine-tuned the open-source Llama-2-7b model (85% accuracy on hard cases, under half a second latency, under $800 cost), before selecting Predibase as their production platform and fine-tuning Llama-3-8b-instruct, using LoRA/LoRAX for parameter-efficient fine-tuning and multi-adapter serving so multiple fine-tuned adapters can share the same GPU infrastructure.

Reported business value

The fine-tuned Llama-3-8b-instruct model in production on Predibase achieved 90% accuracy on the hardest 2% of cases, outperforming GPT-4 and all other experiments, with 0.15-second response times — 30x faster than the GPT-4 experiments — and a 5x cost reduction compared to GPT-4.