SkyHive uses machine learning to power workforce recommendations
SkyHive provides AI-generated labor market intelligence, normalizing global job listings and skill sets across industries and languages. Its data science team uses MLflow on the Databricks Data + AI Platform to orchestrate hundreds of machine learning models that generate workforce and job recommendations, analyzing over 1 petabyte of labor data (150 trillion transactions/day). The lakehouse migration let SkyHive query its full dataset in under 20 minutes (versus 22+ hours to process just 1/100th before) and improved time-to-market for new labor insights applications by 13x.
Overview
SkyHive provides AI-generated labor market intelligence, normalizing global job listings and skill sets across industries and languages. Its data science team uses MLflow on the Databricks Data + AI Platform to orchestrate hundreds of machine learning models that generate workforce and job recommendations, analyzing over 1 petabyte of labor data (150 trillion transactions/day). The lakehouse migration let SkyHive query its full dataset in under 20 minutes (versus 22+ hours to process just 1/100th before) and improved time-to-market for new labor insights applications by 13x.
This entry has 13 published fields tied to exact passages in an immutable source capture.
Inspect the highlighted sourceThe challenge
SkyHive's traditional Hadoop infrastructure struggled with the tremendous amounts of data being processed and analyzed, creating scalability and speed issues, limiting efficacy, and impacting the end-user experience; the company outgrew its Hadoop environment because accessibility to the data was compromised due to data silos, scalability was limited, and the supporting infrastructure was too expensive to operate.
The solution
SkyHive selected Databricks Data + AI Platform over a cloud data warehouse for its multicloud support and ability to unify data for analytics and AI; its data science team uses MLflow to orchestrate hundreds of machine learning models designed to serve workforce recommendations, with Delta Lake and Lakeflow Declarative Pipelines removing data engineering complexity.
Reported business value
SkyHive is now able to access all its data and query it in less than 20 minutes, versus needing at least 22 hours to process just 1/100th of its data set before, and has improved its time-to-market of new labor insights applications by 13x, shortening delivery times from 6-8 months to only 2 and a half weeks.
Sources
Open any source and check the claim yourself — that is the point of the register.
This record was researched and written with AI assistance, and its claims were checked against the sources above. (EU AI Act art. 50 transparency notice.)
Other technology & software entries in the register.
HP crafts marketing campaigns that resonate with customers using Databricks and Uniphore
HP centralized first-party customer data on the Databricks Data + AI Platform with Delta Lake and Unity Catalog, and connected it to Uniphore's HybridCompute for federated query pushdown, cutting campaign setup from 2 weeks to 2 hours and processing 400 million records in seconds.
Building a safer and more sustainable world
Novade, a construction management software company, partnered with Databricks to modernize its analytics and ML infrastructure, moving from a homegrown Apache Airflow stack to the Databricks Data + AI Platform with Delta Lake, Unity Catalog and MLflow. The move reduced total cost of ownership by 60% and supported a 100% increase in new clients, while enabling ML models for predicting incident risk affecting worker safety and project delivery schedules.
Transforming Weather Forecasting with Lakeflow Jobs
AccuWeather migrated from on-premises infrastructure to Databricks and Lakeflow Jobs, working with Datadog for observability, to unify diverse weather data formats and orchestrate 4,500+ weekly jobs. Lakeflow Jobs coordinates the ingestion of multiple weather models, triggers machine learning processes that weight and blend different forecasts, and manages complex job dependencies for reinforcement training workflows used in AccuWeather's proprietary forecasting engine. AccuWeather reports 3x faster dataset development (three months to one month per dataset), a 50% reduction in unactionable alerts, and 50% cost savings on serverless job usage.
AccuWeatherLakeflow Jobs · Delta Sharing · DatadogAdobe brings creativity to life with Databricks
Adobe uses the Databricks Data + AI Platform for end-to-end data management that unifies all data and AI at scale, with 20% faster performance. Databricks equips over 92 teams at Adobe to unify data from financials, sales, products, customers and employees so they can drive personalized experiences across Adobe's digital platforms with AI.
Was this helpful?
Your feedback helps us improve our use case database