Lakeflow Jobs enables automation and collaboration in the energy sector

Overview

Wood Mackenzie, an energy and natural resources consulting firm founded in Edinburgh, used Databricks Lakeflow Jobs to automate ETL pipelines processing 12 billion data points weekly for its Lens data analytics platform. This delivered an 80-90% reduction in processing time and cost savings through workflow automation.

The challenge

Different members of the data team are responsible for different parts of the pipeline, and there is a dependency between the processing stages each team member owns. Without a common workflow, different members of the team would run their notebooks independently, not knowing that failure in their run affected stages downstream.

The solution

Using Databricks Lakeflow Jobs, the team defined a common workstream that the entire team uses, with each stage of the pipeline implemented in a Python notebook run as a job in the main workflow. Going forward, Wood Mackenzie plans to optimize its use of Databricks Lakeflow Jobs to automate machine learning processes such as model training, model monitoring and handling model drift; the firm uses ML to improve its data quality and extract insights to provide more value to its clients.