SCAN Health Plan uses NLP and ML models on Databricks to reduce voluntary disenrollment among senior members

Overview

SCAN Health Plan built an Insights Platform on the Databricks Data + AI Platform to analyze unstructured data such as call center notes and medical charts with NLP. A model that identified members likely to ask about primary care physician reassignment let SCAN intervene early, decreasing disenrollment of newly enrolled members by 3.6%, and a related model reduced disenrollment during annual enrollment by 4.59%.

The challenge

As SCAN scaled its member base, it accumulated an ever-growing mix of structured, unstructured and semi-structured data (call center notes, medical charts, PDF medical documents) sitting around 20-25% of total data, which the team lacked the tools or infrastructure to analyze. This blocked development of an NLP pipeline meant to uncover member behavioral patterns, and conventional analytics acted like an “odometer” rather than surfacing insights to prioritize preventive health.

The solution

SCAN built an Insights Platform on the Databricks Data + AI Platform, using Delta Lake to build reliable data pipelines, Databricks SQL to apply data-warehousing techniques to data lakes, Databricks Unity Catalog with Collibra for data governance and visibility, and Tableau/Power BI for visualization. NLP models detect member behavioral patterns and predict voluntary disenrollment and chronic-condition risk, triggering case management and home-care interventions.

Reported business value

Calling online-enrolled members early to identify the best primary care provider reduced the likelihood new members leave in the future by 3.6%, and an NLP-informed intervention campaign for members at risk during annual enrollment decreased disenrollment of high-risk members by 4.59%, while also reducing the cost of IT operations and increasing SCAN's ability to scale.