Each colour maps a published value to the exact source passage used to support it. Only bounded excerpts are public; administrators can inspect the complete captured source.
Title
Vannevar Labs bolsters defense intelligence with a fine-tuned sentiment analysis model on Databricks
…am leveraged Databricks Model Training to fine-tune their models. Specifically, they fine-tuned Mistral’s 7B parameter model using domain-specific data. This model was chosen for its open source nature and its ability to efficientl…
Description
Defense-tech startup Vannevar Labs used Databricks to fine-tune and deploy a Mistral 7B sentiment analysis model for multilingual news, blog and social media analysis, achieving 76% accuracy versus 64% with GPT-4, a 75% reduction in latency, and a two-week deployment time.
…ce for a safer world 2 weeks to deploy a fully trained sentiment analysis model 76% model accuracy, compared with 64% using GPT-4 75% reduction in model latency Defense-tech startup Vannevar Labs plays a criti…
…ined sentiment analysis model 76% model accuracy, compared with 64% using GPT-4 75% reduction in model latency Defense-tech startup Vannevar Labs plays a critical role in supporting America’…
…Vannevar Labs CUSTOMER STORY Bolstering defense intelligence for a safer world 2 weeks to deploy a fully trained sentiment analysis model 76% model accuracy, compared with 64% using GPT-4 75% reduction in model latenc…
Company
Vannevar Labs
…6% model accuracy, compared with 64% using GPT-4 75% reduction in model latency Defense-tech startup Vannevar Labs plays a critical role in supporting America’s strategic efforts to deter and de-escalate global conflicts. Operating at the forefront of defense technology, the company leverages advance…
Industry
Technology & Software
…ate conflict around the world, visit vannevarlabs.com . Share this post Details Industry : Technology and Software Use Case : Artificial Intelligence Product : Agent Bricks Ready to get started?…
Problem
Before partnering with Databricks, Vannevar Labs struggled with the limitations of commercial models such as GPT-4, which delivered suboptimal accuracy (around 65%) and was not cost-effective, especially given the multilingual complexities of data in Tagalog, Spanish, Russian and Mandarin.
…ngineering, which did not yield satisfactory results for their specific needs. “The best results we could get were around 65% accuracy, and it was overall too expensive for us,” Punma added. “There was also a multilingual problem. We have data in Tagalog,…
…all too expensive for us,” Punma added. “There was also a multilingual problem. We have data in Tagalog, Spanish, Russian and Mandarin, and GPT-4 struggled with lower-resourced languages like Tagalog.” This led them to consider fine-tuning a model using their domain-specific data…
Solution
Vannevar Labs used Databricks to build an end-to-end compound AI system for data ingestion, model fine-tuning and deployment. The team used Databricks Model Training to fine-tune Mistral's 7B parameter open-source model on domain-specific data, chosen for its ability to run efficiently on a single NVIDIA A10 Tensor Core GPU, and used Databricks's Command Line Interface (MCLI) and Python SDK to orchestrate, scale and monitor GPU nodes and container images for training and deployment.
…hieve the accuracy and efficiency required for their critical defense missions. Punma’s team leveraged Databricks Model Training to fine-tune their models. Specifically, they fine-tuned Mistral’s 7B parameter model using domain-specifi…
…cally, they fine-tuned Mistral’s 7B parameter model using domain-specific data. This model was chosen for its open source nature and its ability to efficiently operate on a single NVIDIA A10 Tensor Core GPU. “To meet the real-time demands of our applications, we needed a smaller model t…
…l to fit on a single A10 and have very low real-time latency,” Punma explained. Databricks’s Command Line Interface (MCLI) and Python SDK tools made it easy for Vannevar to orchestrate, scale and monitor the GPU nodes and container images used in model training and deployment. MCLI’s robust capabilities for data ingestion allowed seamless, secure connecti…
Business value
The fine-tuned model achieved an overall F1 score of 76%, an improvement over the 65% accuracy previously achieved with GPT-4, and delivered results faster and more cost-effectively. Latency time was reduced by 75% compared with previous implementations, and the team went from a tutorial to deploying a fully functional, fine-tuned sentiment analysis model within just 2 weeks.
…y to enhance their collection efforts across multiple defense missions quickly. The fine-tuned model achieved an overall F1 score of 76%, an improvement over the 65% accuracy previously achieved with GPT-4, and delivered results faster and more cost-effectively. Latency time was reduced by 75% compared with previous implementations. Accordi…
…ly achieved with GPT-4, and delivered results faster and more cost-effectively. Latency time was reduced by 75% compared with previous implementations. According to Punma, “On all three fronts — accuracy, cost and speed — the fine-…
…Labs to enhance their sentiment analysis capabilities effectively. Punma said, “Within just 2 weeks, we were able to go from a tutorial to deploying a fully functional, fine-tuned sentiment analysis model.” This rapid deployment was a critical success factor, enabling the company to e…
AI capabilities
Natural Language Processing, Large Language Models, AI Model Development & MLOps
…am leveraged Databricks Model Training to fine-tune their models. Specifically, they fine-tuned Mistral’s 7B parameter model using domain-specific data. This model was chosen for its open source nature and its ability to efficientl…
Technology
Databricks Model Training, Mistral 7B, NVIDIA A10 Tensor Core GPU
…hieve the accuracy and efficiency required for their critical defense missions. Punma’s team leveraged Databricks Model Training to fine-tune their models. Specifically, they fine-tuned Mistral’s 7B parameter model using domain-specifi…
…cally, they fine-tuned Mistral’s 7B parameter model using domain-specific data. This model was chosen for its open source nature and its ability to efficiently operate on a single NVIDIA A10 Tensor Core GPU. “To meet the real-time demands of our applications, we needed a smaller model t…
Deployment model
cloud
…l to fit on a single A10 and have very low real-time latency,” Punma explained. Databricks’s Command Line Interface (MCLI) and Python SDK tools made it easy for Vannevar to orchestrate, scale and monitor the GPU nodes and container images used in model training and deployment. MCLI’s robust capabilities for data ingestion allowed seamless, secure connecti…
Deployment options
cloud
…l to fit on a single A10 and have very low real-time latency,” Punma explained. Databricks’s Command Line Interface (MCLI) and Python SDK tools made it easy for Vannevar to orchestrate, scale and monitor the GPU nodes and container images used in model training and deployment. MCLI’s robust capabilities for data ingestion allowed seamless, secure connecti…
Headline outcome
…ined sentiment analysis model 76% model accuracy, compared with 64% using GPT-4 75% reduction in model latency Defense-tech startup Vannevar Labs plays a critical role in supporting America’…
Use case type
Analytics augmentation
…t vannevarlabs.com . Share this post Details Industry : Technology and Software Use Case : Artificial Intelligence Product : Agent Bricks Ready to get started? Try Databricks for free Learn more…
