NOL UNIVERSE builds unified data lakehouse with Databricks, cutting batch computing costs by 77%

Overview

NOL UNIVERSE, a Korean travel, leisure, and culture platform operating the NOL, NOL Interpark Tours, and NOL Tickets brands, migrated approximately 2,500 Hive-based queries and its Apache Airflow/NiFi pipelines to the Databricks Platform, consolidating disparate BI tools into a single environment and giving each business domain a self-service analytics environment governed by Unity Catalog. The customer service team also used Databricks machine learning to automate categorization of CS consultation history. The migration was completed in two months, reducing time to complete batch aggregation by 66%, decreasing batch computing costs by 77%, and increasing data availability time by 27%.

The challenge

NOL UNIVERSE's existing data environment relied on multiple databases (MySQL, SQL Server), Hive and Spark on Amazon EMR, and a combination of Apache Airflow and Apache NiFi for pipelines, with several separate BI tools (Redash, Tableau). As accumulated data and batch processing time grew, the separation of the data processing engine and scheduling system made operations inefficient, causing delays in tracing and responding to batch failures, and different cloud accounts and permission schemes for each service added difficulty. The data team's full responsibility for all data products created a bottleneck during peak workload.

The solution

NOL UNIVERSE adopted the Databricks Data + AI Platform, migrating approximately 2,500 existing Hive-based queries and its Apache Airflow/NiFi pipelines to Databricks, and consolidating disparate BI tools into one platform. Each business domain was given a self-service analytics environment governed by Unity Catalog, with SSO integration unifying user authentication. The customer service team used Databricks machine learning to automate categorization of CS consultation history. The migration ran as a proof of concept in August 2024, followed by an intensive transition and validation process from September through November, completing in two months.

Reported business value

The migration reduced time to complete batch aggregation by 66%, decreased batch computing costs by 77%, and increased data availability time by 27%. The customer service categorization task that used to take over a day can now be completed in less than two hours.