Each colour maps a published value to the exact source passage used to support it. Only bounded excerpts are public; administrators can inspect the complete captured source.
Title
University Hospital Halle runs a locally hosted Falcon-7B RAG chatbot for internal document retrieval
…ernal document library at the University Hospital Halle via a chatbot. Methods: The system is hosted on a local Linux server equipped with an Nvidia H100 GPU and utilizes the Falcon-7B-Instruct model. Internal documents - primarily SOPs, procedural guidelines and forms - were cle…
Description
A proof-of-concept at University Hospital Halle (Saale) hosts Falcon-7B-Instruct with retrieval-augmented generation on a local Linux server with one Nvidia H100 GPU, over internal SOPs, guidelines and forms. In a one-month test phase it handled 287 user queries; 75% of responses returned in under five seconds.
…luation questions and (3) a technically analysis of the response time. Results: During the test phase the system handled 287 user queries, receiving 72 feedback entries (25.01% response rate). Feedback on document rel…
…3.89 for retrieved documents. Technical performance showed acceptable latency, with 75% of responses returned in under five seconds. First queries in a session averaged 3.71s, while follow-ups averaged with 4.28…
Company
University Hospital Halle (Saale)
…artin-Luther-University Halle-Wittenberg, Halle (Saale), Germany Thomas Weber - University Hospital Halle (Saale), Department of Internal Medicine IV, Hematology and Oncology, Halle (Saale), Ge…
Country
Germany
…l Data Science, Faculty of Medicine, Martin-Luther-University Halle-Wittenberg, Halle (Saale), Germany Jan Bossenz - Junior Research Group (Bio-)Medical Data Science, Faculty of Medi…
Industry
Healthcare
…n-Luther-University Halle-Wittenberg, Halle (Saale), Germany Text Introduction: The integration of large language models (LLMs) into healthcare, with its strict data protection and security requirements, poses particular ch…
Problem
Healthcare has strict data protection and security requirements, so powerful proprietary models such as ChatGPT are not suitable even though they are increasingly used informally. The hospital wanted a way to interact with its extensive locally stored internal document library.
…Ms) into healthcare, with its strict data protection and security requirements, poses particular challenges, which is why powerful proprietary models such as ChatGPT are not suitable, even if they are increasingly being used informally. To close this gap, this project presents a locally hosted open-source LLM and…
…lly hosted open-source LLM and retrieval-augmented generation (RAG) system that enables interaction with a locally stored, extensive internal document library at the University Hospital Halle via a chatbot. Methods: The system is hosted on a local Linux server equipped with an Nvidia…
Solution
A locally hosted open-source LLM and retrieval-augmented generation (RAG) system on a local Linux server with an Nvidia H100 GPU, using the Falcon-7B-Instruct model. Internal documents, primarily SOPs, procedural guidelines and forms, were cleaned, contextually chunked and embedded with the jina-embeddings-v3 model, and the system retrieves documents using combined cosine similarity and BM25 scoring through a chatbot.
…cleaned, contextually chunked, and embedded using the jina-embeddings-v3 model. The system retrieves documents based on combined cosine similarity and BM25 scoring. Evaluation involved (1) subjective user feedback collected during a one-month t…
Business value
In a one-month test phase the proof-of-concept system handled 287 user queries, with 75% of responses returned in under five seconds. Experts rated textual answers 3.72 and retrieved documents 3.89 on a scale from one to five, and user feedback was mixed, which the authors attribute to the early prototype status. The authors conclude the system already offers a useful alternative to basic keyword search.
…evaluation using the test dataset yielded an average helpfulness rating of 72%. On a scale from one (worse) to five (best) expert evaluation reached a score of 3.72 for textual answers and 3.89 for retrieved documents. Technical performance showed acceptable latency, with 75% of responses returned…
…LMs can yield useful results when coupled with well-designed retrieval systems. Although user feedback was mixed, it reflected the early prototype status and the subjective nature of expectations. Document retrieval quality was consi…
AI capabilities
Generative AI, Conversational AI
…increasingly being used informally. To close this gap, this project presents a locally hosted open-source LLM and retrieval-augmented generation (RAG) system that enables interaction with a locally stored, extensive internal document lib…
Technology
Falcon-7B-Instruct, jina-embeddings-v3, Nvidia H100
…e system is hosted on a local Linux server equipped with an Nvidia H100 GPU and utilizes the Falcon-7B-Instruct model. Internal documents - primarily SOPs, procedural guidelines and forms - were cl…
…SOPs, procedural guidelines and forms - were cleaned, contextually chunked, and embedded using the jina-embeddings-v3 model. The system retrieves documents based on combined cosine similarity and BM25 sc…
…ital Halle via a chatbot. Methods: The system is hosted on a local Linux server equipped with an Nvidia H100 GPU and utilizes the Falcon-7B-Instruct model. Internal documents - primarily SOPs,…
Deployment model
on-premise
…gulate document access by various individuals. Overall, the project underscores the feasibility and potential of secure, on-premise AI tools in clinical environments. Jan Kirchhoff and Fabian Berns are employed by medicalvalues GmbH, which provi…
Deployment options
On-premise
…library at the University Hospital Halle via a chatbot. Methods: The system is hosted on a local Linux server equipped with an Nvidia H100 GPU and utilizes the Falcon-7B-Instruct model. Int…
Headline outcome
…3.89 for retrieved documents. Technical performance showed acceptable latency, with 75% of responses returned in under five seconds. First queries in a session averaged 3.71s, while follow-ups averaged with 4.28…
Use case type
Knowledge search
…matik, Biometrie und Epidemiologie e.V. Implementing Local Open-Source LLMs for Chatbot-Based Document Retrieval in a University Hospital 70. Jahrestagung der Deutschen Gesellschaft für Medizinische Informatik, Biomet…
