Stadt Bern proof of concept: open-source RAG chatbot on Llama-3.3-70B answers staff questions on legal documents

Overview

The Institute Public Sector Transformation at Bern University of Applied Sciences built a proof-of-concept chatbot for the City of Bern's resident services. It uses retrieval-augmented generation over a knowledge base of 296 documents with Streamlit, Haystack, Snowflake Arctic Embed 2.0, a reranker and Llama-3.3-70B-Instruct. It shows sources, supports follow-up questions, and all data can be processed on the city's own servers. Evaluated qualitatively by a lawyer and with automated metrics.

The challenge

The City of Bern needed a solution to help staff answer questions on complex legal documents, for example in the foreigners and migration area. Existing proprietary AI solutions are restricted by data protection concerns because sensitive documents may not be processed on external servers.

The solution

The Institute Public Sector Transformation at Bern University of Applied Sciences developed a retrieval-augmented generation service over a knowledge base of 296 documents (3,932 pages), using Streamlit as frontend and Haystack as AI backend. Three models are used: the Snowflake Arctic Embed 2.0 embedding model, the BAAI/bge-reranker-v2-m3 reranker and the Llama-3.3-70B-Instruct language model. Queries are processed lexically (BM25) and semantically, the chatbot supports follow-up questions and shows its sources.

Reported business value

This was a proof of concept, not a productive deployment. In a qualitative evaluation by a lawyer, answers were mostly correct and sources precise. In automated evaluation the setup reached a recall of 0.73 and nDCG of 0.46, and Llama-3.3-70B-Instruct scored 0.92 for answer relevancy and 0.92 for faithfulness. The project states that the open-source approach allows all data to be processed on the city's own servers, with low, predictable operating costs.