Kantar rebuilds its Worldpanel Plus data processing platform on Databricks for faster brand insights

Overview

Kantar replaced three in-house architectures with a new data processing platform on the Databricks Data + AI Platform on Azure to process 3 million+ consumer purchase receipts per month for its Worldpanel Plus research service, using Unity Catalog, Delta Sharing, Spark Declarative Pipelines and MLflow, cutting the time to reprocess five years of historical data from several months to two days.

The challenge

Kantar's previous SQL Server-based platform could process new receipts but struggled with reprocessing five years of historical purchase data, which was crucial for delivering trend insights such as market share changes; the legacy SAS system was also too rigid, complex and costly to maintain, making machine learning models simply not possible.

The solution

Kantar built a new data processing platform for its Worldpanel Plus research service on the Databricks Data + AI Platform on Azure. Purchase receipts scanned via OCR flow into the platform, which uses Unity Catalog for governance, Delta Sharing to deliver data to clients, Spark Declarative Pipelines for streaming, and MLflow with Databricks Model Serving to develop and deploy machine learning models for data cleaning and insight generation.

Reported business value

Kantar can now process five years of data in two days instead of potentially several months, processes more than 3 million receipts per month, deployed a new client data-sharing capability in just one week, and was able to pivot to real-time daily reporting for clients during the COVID-19 pandemic.