Each colour maps a published value to the exact source passage used to support it. Only bounded excerpts are public; administrators can inspect the complete captured source.
Title
Vanderbilt University applies NLP on Databricks to analyze decades of TV news archive for research
Enhancing data analysis in academic research through “Radical Collaboration” | Databricks Skip to main content Login Why Databricks Discover For App Develop…
Description
The Jean and Alexander Heard Libraries at Vanderbilt University transcribed their Vanderbilt Television News Archive and deployed the Databricks Data + AI Platform, using Unity Catalog and Spark Declarative Pipelines, to let researchers run natural language processing and machine learning experiments pairing the archive with datasets like polling data and public policy documents, including studying the impact of presidential executive orders on public discourse. Since implementation, citations of Vanderbilt scholarship using the archive have increased 2.7x.
…r as much Vanderbilt scholarship to be as open access as possible,” Shaw said. “Now, Vanderbilt is being cited approximately 2.7 times more often than before.” This underscores the broader impact of Vanderbilt’s credibility on the global…
Company
Vanderbilt University
…earch and policy impact 2.7x Increase in research citations ~8PB In data volume The Jean and Alexander Heard Libraries at Vanderbilt University provide access to a world of resources that academic institutions use to conduct research and teach in the classroom.…
Industry
Education
…expertise needed to fuel academic discovery and collaboration. Share this post Details Industry : Education , Public Sector Use Case : Data Science Cloud : AWS Product : Delta Lake , Spark Declarative Pi…
Problem
The Heard Libraries lacked the capacity and full-text transcripts to let scholars run natural language processing tools on the Vanderbilt Television News Archive, and accessing and manipulating this large-scale data was cumbersome with limited capability to handle complex queries spanning multiple data types and time periods.
…r of their extensive Television News Archive to support diverse research needs. Scholars sought to run their natural language processing tools on the data, but the Heard Libraries did not have the capacity — or the full-text transcripts — to allow this level of analysis. One prominent use case involved researching the impact of presidential executiv…
Solution
With a gift funding transcription of all video content, Vanderbilt deployed the Databricks Data + AI Platform to provision secure researcher access, using Unity Catalog for governance and Spark Declarative Pipelines for automatic ingestion of new data, enabling researchers to run NLP techniques and machine learning experiments pairing the TV News Archive with other datasets.
…or of the Vanderbilt Television News Archive at Vanderbilt University, stated, “With a gift from Suzanne Perot McGee, Patrick K. McGee and their family, we were able to transcribe all of the video content and deploy Databricks to provision secure access to our researchers. This has allowed them to run machine learning research and experiments, pairing…
Technology
Unity Catalog, Spark Declarative Pipelines, Delta Lake
…nt data management, crucial for maintaining data integrity and confidentiality, through features such as Unity Catalog for governance and Spark Declarative Pipelines for automatic ingestion of new data. These tools facilitate seamless collaboration among researchers and support th…
Deployment model
cloud
…ess, analyze and understand context from TV broadcasts and other data at scale, they chose the Databricks Data + AI Platform. Vanderbilt University has since transformed their delivery of historical infor…
Deployment options
cloud
…expertise needed to fuel academic discovery and collaboration. Share this post Details Industry : Education , Public Sector Use Case : Data Science Cloud : AWS Product : Delta Lake , Spark Declarative Pipelines , Unity Catalog Ready to get…
Headline outcome
…r as much Vanderbilt scholarship to be as open access as possible,” Shaw said. “Now, Vanderbilt is being cited approximately 2.7 times more often than before.” This underscores the broader impact of Vanderbilt’s credibility on the global…
Business value
Vanderbilt is now cited approximately 2.7 times more often than before, usage of the Vanderbilt Television News Archive grew from thousands of users toward a projected hundreds of thousands, and researchers without deep data science expertise can now run advanced NLP analysis that was previously not possible.
…he libraries have reported a dramatic increase in the usage of their resources. Before implementation of the Databricks Data + AI Platform, the Vanderbilt Television News Archive was utilized by thousands. Shaw now believes it’s poised to engage hundreds of thousands. “Databricks has allowed for as much Vanderbilt scholarship to be as open access…
AI capabilities
Natural Language Processing, Machine Learning
…eo content and deploy Databricks to provision secure access to our researchers. This has allowed them to run machine learning research and experiments, pairing the TV News Archive with other datasets to explore complex issues like the effects of executive orders on public percep…
Use case type
Analytics augmentation
…ast news to derive meaningful insights into public discourse and policy impact. Databricks democratized data access and the use of advanced NLP tools among researchers, including those without deep technical expertise in data science. “The platform’s intuitive interface allows researchers to focus more on the dis…
