Each colour maps a published value to the exact source passage used to support it. Only bounded excerpts are public; administrators can inspect the complete captured source.
Title
Lakeflow Jobs enables automation and collaboration in the energy sector
…y Databricks Customer Stories / Wood Mackenzie PRODUCT SPOTLIGHT: Lakeflow Jobs Lakeflow Jobs enables automation and collaboration in the energy sector 12 Billion Data points processed each week 80-90% Reduction in processing time…
Description
Wood Mackenzie, an energy and natural resources consulting firm founded in Edinburgh, used Databricks Lakeflow Jobs to automate ETL pipelines processing 12 billion data points weekly for its Lens data analytics platform. This delivered an 80-90% reduction in processing time and cost savings through workflow automation.
…ction in processing time Cost Savings In operations through workflow automation Wood Mackenzie offers customized consulting and analysis for a wide range of clients in the energy and natural resources sectors. Founded in Edinburgh, the company first cultivated deep expertise in upstream oil and gas, then broadened its focus to deliver detailed insight for every interconnected sector of the energy, chemicals, metals and mining industries. Today it sees itself playing an important role in the transition to a more sust…
…tself playing an important role in the transition to a more sustainable future. Using Databricks Lakeflow Jobs to automate ETL pipelines helps Wood Mackenzie ingest and process massive amounts of data. Using a common workflow provided higher visibility to engineering team members,…
…and sensors used to monitor energy creation, oil and gas production, and more. Those data sources update about 12 billion data points every week that must be ingested, cleaned and processed as part of the input for the Lens platform. Yanyan Wu, Vice President of Data at Wood Mackenzie, manages a team of big data…
…d collaboration in the energy sector 12 Billion Data points processed each week 80-90% Reduction in processing time Cost Savings In operations through workflow automation Wood Mackenzie offers cu…
…12 Billion Data points processed each week 80-90% Reduction in processing time Cost Savings In operations through workflow automation Wood Mackenzie offers customized consulting and analysis for a wide range of cl…
Company
Wood Mackenzie
…rity and Trust Ready to get started? Get a Demo Login Contact Us Try Databricks Customer Stories / Wood Mackenzie PRODUCT SPOTLIGHT: Lakeflow Jobs Lakeflow Jobs enables automation and collabora…
Industry
Energy
…flexibility to deliver the insights our clients need.” Share this post Details Industry : Energy , Industrials Product : Lakeflow Jobs Ready to get started? Try Databricks for free Learn mor…
Problem
Different members of the data team are responsible for different parts of the pipeline, and there is a dependency between the processing stages each team member owns. Without a common workflow, different members of the team would run their notebooks independently, not knowing that failure in their run affected stages downstream.
…tured or unstructured and may be in the form of PDFs or even handwritten notes. Different members of the data team are responsible for different parts of the pipeline, and there is a dependency between the processing stages each team member owns. Using Databricks Lakeflow Jobs , the team defined a common workstream that the…
…t of the pipeline that originated the problem makes fixing issues much faster. “Without a common workflow, different members of the team would run their notebooks independently, not knowing that failure in their run affected stages downstream,” says Meng Zhang, Principal Data Analyst at Wood Mackenzie. “When trying to re…
Solution
Using Databricks Lakeflow Jobs, the team defined a common workstream that the entire team uses, with each stage of the pipeline implemented in a Python notebook run as a job in the main workflow. Going forward, Wood Mackenzie plans to optimize its use of Databricks Lakeflow Jobs to automate machine learning processes such as model training, model monitoring and handling model drift; the firm uses ML to improve its data quality and extract insights to provide more value to its clients.
…and there is a dependency between the processing stages each team member owns. Using Databricks Lakeflow Jobs , the team defined a common workstream that the entire team uses. Each stage of the pipeline is implemented in a Python notebook, which is run as a job in the main workflow. Each team member can now see exactly what code is running on each stage, making…
…nch and the production workflow is automatically updated with the latest code.” Going forward, Wood Mackenzie plans to optimize its use of Databricks Lakeflow Jobs to automate machine learning processes such as model training, model monitoring and handling model drift. The firm uses ML to improve its data quality and extract insights to provide more value to its clients. “Our mission is to transform how we power the planet,” Wu says. “Our clients in…
AI capabilities
Predictive Analytics
…ng processes such as model training, model monitoring and handling model drift. The firm uses ML to improve its data quality and extract insights to provide more value to its clients. “Our mission is to transform how we power the planet,” Wu says. “Our clients in…
Technology
Lakeflow Jobs, Apache Spark
…ghts our clients need.” Share this post Details Industry : Energy , Industrials Product : Lakeflow Jobs Ready to get started? Try Databricks for free Learn more about our product Talk…
…put data for Lens. The team is leveraging the Databricks Data + AI Platform and uses Apache Spark™ for parallel processing, which provides greater performance and scalability benefits compared to an ear…
Deployment model
cloud
…als that build and maintain the ETL pipeline that provides input data for Lens. The team is leveraging the Databricks Data + AI Platform and uses Apache Spark™ for parallel processing, which provides greater performance and scalability benefits compared to an ear…
Deployment options
cloud
…als that build and maintain the ETL pipeline that provides input data for Lens. The team is leveraging the Databricks Data + AI Platform and uses Apache Spark™ for parallel processing, which provides greater performance and scalability benefits compared to an ear…
Headline outcome
…d collaboration in the energy sector 12 Billion Data points processed each week 80-90% Reduction in processing time Cost Savings In operations through workflow automation Wood Mackenzie offers cu…
