Abnormal Security detects email attacks with behavioral AI on Databricks

Overview

Abnormal Security migrated from a legacy Hadoop system to the Databricks Data + AI Platform on AWS, using Delta Lake and Databricks SQL to stream threat signals from Kinesis Firehose and power behavioral AI models analyzing over 50,000 signals to detect email attacks, achieving a 20% reduction in successful email attacks, a 40% reduction in infrastructure costs, and a 30%+ increase in productivity.

The challenge

Abnormal Security's legacy Hadoop system with long-running AWS EMR clusters wasn't going to scale efficiently with rapidly increasing demand, and engineers were spending too much time managing Spark infrastructure instead of building pipelines that would make the product better; a shared Jupyter notebook server also slowed processing for everyone when heavy applications ran.

The solution

Abnormal Security migrated to the Databricks Data + AI Platform on AWS, using Delta Lake to stream threat signals from Kinesis Firehose into near real-time storage and Databricks SQL for dashboards, powering behavioral AI models that analyze over 50,000 signals to detect and remediate email attacks.

Reported business value

Abnormal Security achieved a 20% reduction in successful email attacks, a 40% reduction in infrastructure costs, and 30%+ productivity gains for its data science and data engineering teams, while processing thousands of emails per second.