Transforming Weather Forecasting with Lakeflow Jobs

Overview

AccuWeather migrated from on-premises infrastructure to Databricks and Lakeflow Jobs, working with Datadog for observability, to unify diverse weather data formats and orchestrate 4,500+ weekly jobs. Lakeflow Jobs coordinates the ingestion of multiple weather models, triggers machine learning processes that weight and blend different forecasts, and manages complex job dependencies for reinforcement training workflows used in AccuWeather's proprietary forecasting engine. AccuWeather reports 3x faster dataset development (three months to one month per dataset), a 50% reduction in unactionable alerts, and 50% cost savings on serverless job usage.

The challenge

Faced with the challenge of ingesting, processing and storing immense volumes of diverse weather data at scale, AccuWeather's on-premises infrastructure forced regular purging of valuable data, preventing historical analysis of model performance, and numerous data sources without a single source of truth created expensive data silos and redundant processing costs.

The solution

AccuWeather migrated to Databricks and selected Lakeflow Jobs as the primary workflow engine, working with Datadog for observability, to automate manual planning and validation processes. Lakeflow Jobs orchestrates seamless dependencies across 4,500+ weekly jobs, coordinates the ingestion of multiple weather models, triggers machine learning processes that weight and blend different forecasts, and manages complex job dependencies for reinforcement training workflows used in AccuWeather's proprietary forecasting engine.

Reported business value

3x Faster dataset development, a 50% reduction in unactionable alerts, and 50% cost savings on serverless job usage; one critical job converted entirely to serverless Lakeflow Jobs cut usage costs in half.