Each colour maps a published value to the exact source passage used to support it. Only bounded excerpts are public; administrators can inspect the complete captured source.
Title
Carvana rebuilds its customer communications data pipeline with Databricks Spark Declarative Pipelines
…ricks Customer Stories / Carvana PRODUCT SPOTLIGHT: Spark Declarative Pipelines Carvana turns to Databricks Spark Declarative Pipelines and streaming to uplevel CX $500k Saved in data warehousing costs Carvana is an online used car retailer ba…
Description
Carvana's Next Generation Communication Platform team migrated from Google BigQuery and Amazon EMR to the Databricks Data + AI Platform, using Spark Declarative Pipelines, Databricks SQL Serverless and Delta Lake to manage a 3-billion-record dataset powering NLP-driven chatbot and customer communications, saving approximately $500,000 a year in data warehousing costs.
…At the time of writing, the team consists of five data and analytics engineers. The biggest data set they manage has about 3 billion records, a number that is still growing today. The team initially streamed its conversation and AI data into Google BigQuery,…
…more to improve CX Reducing full table scans and data extraction from BigQuery has saved Carvana approximately $500,000 a year in data warehouse costs. Meanwhile, the lakehouse provides a foundation for Carvana’s data scientists a…
Company
Carvana
…ive Pipelines and streaming to uplevel CX $500k Saved in data warehousing costs Carvana is an online used car retailer based in Arizona. Founded in 2012, the company’s mission is to change how people buy and sell car…
Industry
Retail
…nd SQL,” says Taylor. “Databricks tools can help get us there.” Share this post Details Industry : Digital and AI-Native Business , Retail and Consumer Goods Use Case : Data Engineering Cloud : AWS , Azure Product : Delta Lake , Spark Declarative Pipelines Ready to get started? Try Databricks for free Learn more about our product Talk…
Problem
Carvana's NGCP team streamed conversation and AI data into Google BigQuery, which limited how engineers could partition and optimize query tables, made deduping large data frames slow and caused full recomputation, surfaced data-quality issues only through downstream dashboards, and lacked automatic data availability as experiment campaigns ran, all while producing files too numerous to ship efficiently to the warehouse.
…could partition and optimize query tables. Data quality was another challenge. Engineers needed to dedupe in the pipeline, but distinct calls on large data frames were slow and caused recomputation on the entire data set. Engineers were often only aware of such issues once they surfaced much later th…
Solution
Carvana's NGCP team moved to the Databricks Data + AI Platform, using Spark Declarative Pipelines as a single entry point for streaming and batch jobs, dependency orchestration, data quality checks and error handling, plus Databricks SQL Serverless and Delta Lake for low-latency real-time analytics, to train advanced NLP models that power chatbot and human-advocate customer communications.
…ized Spark Structured Streaming and made it much more accessible for everyone.” Taylor’s team now uses Spark Declarative Pipelines as a single-entry point for streaming and batch jobs, dependency orchestration, data quality, and error handling, enabling them to build scalable and testable pipelines under a data medallion…
Business value
Reducing full table scans and data extraction from BigQuery saved Carvana approximately $500,000 a year in data warehousing costs, while the lakehouse let data scientists and analysts run ad hoc analysis and regression models directly on near real-time data without waiting for cluster spin-up.
…h it.” Lakehouse architecture empowers data scientists to do more to improve CX Reducing full table scans and data extraction from BigQuery has saved Carvana approximately $500,000 a year in data warehouse costs. Meanwhile, the lakehouse provides a foundation for Carvana’s data scientists an…
AI capabilities
Natural Language Processing, Conversational AI
…tbot, human advocates and emails. The team uses AI for initial interactions and trains advanced natural language processing (NLP) models to continuously improve customer experience and speed to answer. At the time of writing, the team consists of five data and analytics engineers…
…customer communications and AI, providing a platform where potential customers interact with Carvana via chatbot, human advocates and emails. The team uses AI for initial interactions and trains advanced natural language…
Technology
Spark Declarative Pipelines, Delta Lake, Databricks SQL
…If an event is delayed, we get real-time alerts.” At the data warehouse level, Carvana uses Databricks SQL Serverless and Delta Lake, which improve speed for real-time analytics use cases where Carvana data engineers need accurate data with very low latency. “Going s…
Deployment options
cloud
…nd SQL,” says Taylor. “Databricks tools can help get us there.” Share this post Details Industry : Digital and AI-Native Business , Retail and Consumer Goods Use Case : Data Engineering Cloud : AWS , Azure Product : Delta Lake , Spark Declarative Pipelines Ready to get started? Try Databricks for free Learn more about our product Talk…
Headline outcome
…more to improve CX Reducing full table scans and data extraction from BigQuery has saved Carvana approximately $500,000 a year in data warehouse costs. Meanwhile, the lakehouse provides a foundation for Carvana’s data scientists a…
Use case type
Conversational assistant
…customer communications and AI, providing a platform where potential customers interact with Carvana via chatbot, human advocates and emails. The team uses AI for initial interactions and trains advanced natural language…
