Observe.AI cuts machine learning costs over 50% with load-testing framework on AWS

Overview

Observe.AI, a conversation intelligence platform using a 30-billion-parameter contact center LLM to analyze customer interactions, developed and open-sourced the One Load Audit Framework (OLAF), integrated with Amazon SageMaker, to automatically identify bottlenecks and performance issues in its ML services and predict data load capacity. By fine-tuning SageMaker instance sizes with OLAF, Observe.AI cut ML deployment costs by over 50%, reduced infrastructure sizing time from one week to a few hours, and enabled on-demand scaling to support a 10x increase in data load.

The challenge

Observe.AI's ML engineers struggled to accurately predict whether its ML system could handle a tenfold increase in data load when moving models from research to production, while needing to manage latency and control costs.

The solution

Observe.AI built and open-sourced the One Load Audit Framework (OLAF), integrated with Amazon SageMaker, Amazon SQS, and Amazon SNS, to load-test ML services and identify bottlenecks, latency, and throughput under static and dynamic data loads.

Reported business value

Using OLAF, Observe.AI cut ML deployment costs by over 50%, reduced the time to determine the right instance configuration from about a week to a few hours, and can now scale to support a tenfold increase in data load.