DDI helps companies develop exceptional leaders with Databricks

Overview

Leadership development firm DDI used Databricks Notebooks, DSPy prompt optimization and fine-tuned Llama3-8b models to automate scoring of behavioral leadership simulations, cutting simulation report generation from 48 hours to 10 seconds, doubling recall score, and improving F1 score by 13%.

The challenge

DDI relied on trained human assessors to score behavioral simulation responses, a process that typically took 24 to 48 hours due to the depth of evaluation required, and faced ML deployment challenges including hardware orchestration, infrastructure management, scaling, data privacy and coordinating multiple vendors.

The solution

DDI used the Databricks Data + AI Platform to develop and deploy ML models for automating behavioral simulation analysis. Using Databricks Notebooks, the team experimented with prompt engineering on OpenAI's GPT-4 (few-shot learning, chain-of-thought prompting, and self-ask prompts), optimized prompts with DSPy, and instruction-fine-tuned a Llama3-8b model. MLflow tracked experiments and LLM operations, and Unity Catalog provided model governance and access control for models deployed as endpoints with Model Serving.

Reported business value

The implementation of ML models has drastically reduced the simulation report delivery time from 48 hours to just 10 seconds. Prompt optimization with DSPy improved the recall score from 0.43 to 0.98, a 100% increase, and the instruction fine-tuned Llama3-8b achieved an F1 score of 0.86, a 13% increase over the baseline score of 0.76.