GDS pilots GOV.UK Chat, an AI assistant, with more than 10,000 users, reaching 90% answer accuracy

Overview

The UK Government Digital Service ran two public pilots of GOV.UK Chat over 18 months; more than 10,000 users asked 26,000 questions about government services. Answers draw exclusively on guidance published on GOV.UK. GDS reports answer accuracy rising from 76% to 90%, 73% of surveyed app users finding it useful, an 88% answer rate for in-scope questions and an average 10.7 seconds per answer; the latest version uses Amazon Bedrock and Anthropic Claude models.

The challenge

GDS wanted to test whether an AI assistant could give users a faster way to get answers about government services, from tax to benefits to visas, while measuring accuracy, reliability, speed, safety and user trust.

The solution

GOV.UK Chat is an AI assistant that exclusively draws on guidance published on GOV.UK and, by design, does not answer outside the scope of government guidance. It can ask clarifying questions when a user's question is ambiguous. The latest version is powered by Amazon's Bedrock platform and Anthropic's Claude models, and is built so that new models can be adopted as they become available. Safety guardrails are in place against jailbreak attempts.

Reported business value

GDS reports answer accuracy rising from 76% at the earliest benchmark to 90% across all topics, 73% of surveyed GOV.UK app users finding Chat useful and 64% satisfied, an 88% answer rate for in-scope questions, and an average of 10.7 seconds per answer. The 508 jailbreak attempts during the pilots were all prevented by the guardrails, according to GDS. These are self-reported pilot results.