Marsh McLennan saves over 1 million hours with agentic AI assistant LenAI

Overview

Professional services firm Marsh McLennan built LenAI, an AI assistant now used by its roughly 90,000 employees for around 20 million requests annually. To fix intent-recognition problems in the original GPT-3.5-based assistant, Marsh McLennan worked with Predibase to fine-tune a Llama-3.1-8B-Instruct model using LoRA and Turbo LoRA, improving accuracy by 10-12% over GPT-3.5 and 7% over GPT-4o-mini, and cutting response time below 4 seconds. Since its December 2023 launch, LenAI has handled over 25 million cumulative queries and saved more than 1 million hours of team time in its first year.

The challenge

Marsh McLennan's original AI assistant, LenAI, was built on GPT-3.5 and occasionally faced difficulties with intent recognition — a crucial capability for agentic AI — sometimes misinterpreting the correct tool based on a user's query; for example, when users requested simple tasks like drafting emails, the model would return documents containing the keyword 'email' rather than launching the email client.

The solution

To reach the next level of accuracy and drive task automation with higher precision, Marsh McLennan worked with Predibase to fine-tune a Llama-3.1-8B-Instruct model using low-rank adaptation (LoRA) and Predibase's Turbo LoRA, deploying on a GPU that supports FP8 quantization.

Reported business value

The fine-tuned Llama-3.1-8B-Instruct model improved accuracy by 10-12% over GPT-3.5 and 7% over GPT-4o-mini, and reduced round-trip request time to below 4 seconds. Since its December 2023 launch, LenAI has supported over 25 million cumulative queries across the organization and saved at least 1 million hours of team time in its first year alone; the company now handles around 20 million requests annually, with upwards of 90,000 people using it.