Carvana rebuilds its customer communications data pipeline with Databricks Spark Declarative Pipelines

Overview

Carvana's Next Generation Communication Platform team migrated from Google BigQuery and Amazon EMR to the Databricks Data + AI Platform, using Spark Declarative Pipelines, Databricks SQL Serverless and Delta Lake to manage a 3-billion-record dataset powering NLP-driven chatbot and customer communications, saving approximately $500,000 a year in data warehousing costs.

The challenge

Carvana's NGCP team streamed conversation and AI data into Google BigQuery, which limited how engineers could partition and optimize query tables, made deduping large data frames slow and caused full recomputation, surfaced data-quality issues only through downstream dashboards, and lacked automatic data availability as experiment campaigns ran, all while producing files too numerous to ship efficiently to the warehouse.

The solution

Carvana's NGCP team moved to the Databricks Data + AI Platform, using Spark Declarative Pipelines as a single entry point for streaming and batch jobs, dependency orchestration, data quality checks and error handling, plus Databricks SQL Serverless and Delta Lake for low-latency real-time analytics, to train advanced NLP models that power chatbot and human-advocate customer communications.

Reported business value

Reducing full table scans and data extraction from BigQuery saved Carvana approximately $500,000 a year in data warehousing costs, while the lakehouse let data scientists and analysts run ad hoc analysis and regression models directly on near real-time data without waiting for cluster spin-up.