Vannevar Labs bolsters defense intelligence with a fine-tuned sentiment analysis model on Databricks

Overview

Defense-tech startup Vannevar Labs used Databricks to fine-tune and deploy a Mistral 7B sentiment analysis model for multilingual news, blog and social media analysis, achieving 76% accuracy versus 64% with GPT-4, a 75% reduction in latency, and a two-week deployment time.

The challenge

Before partnering with Databricks, Vannevar Labs struggled with the limitations of commercial models such as GPT-4, which delivered suboptimal accuracy (around 65%) and was not cost-effective, especially given the multilingual complexities of data in Tagalog, Spanish, Russian and Mandarin.

The solution

Vannevar Labs used Databricks to build an end-to-end compound AI system for data ingestion, model fine-tuning and deployment. The team used Databricks Model Training to fine-tune Mistral's 7B parameter open-source model on domain-specific data, chosen for its ability to run efficiently on a single NVIDIA A10 Tensor Core GPU, and used Databricks's Command Line Interface (MCLI) and Python SDK to orchestrate, scale and monitor GPU nodes and container images for training and deployment.

Reported business value

The fine-tuned model achieved an overall F1 score of 76%, an improvement over the 65% accuracy previously achieved with GPT-4, and delivered results faster and more cost-effectively. Latency time was reduced by 75% compared with previous implementations, and the team went from a tutorial to deploying a fully functional, fine-tuned sentiment analysis model within just 2 weeks.