Air France-KLM's EyeLevel RAG chatbot hits 96.2% accuracy on airline policy questions, beating its 60% target

Overview

Air France/KLM identified airline customer service as a prime use case for generative AI, with a goal of a chatbot trained on thousands of policy documents that could help customer service agents answer traveler questions in seconds. In a three-month test, EyeLevel.ai ingested PDF documents about KLM and Air France services into its retrieval-augmented generation platform; Air France/KLM experts wrote questions, reviewed responses and scored accuracy against the source documents. After an optimization process tuning style, formatting and tone, the assistant reached 96.2% accuracy against a 60% proof-of-concept target.

The challenge

Airline customer service agents need to understand thousands of complicated policies across hundreds of locations; basic training takes nine months or more and travelers endure long wait times. The knowledge base is filled with visually complex documents (tables, diagrams, decision trees, software screenshots), and a question as simple as the luggage price involves nearly a dozen rules covering class, fare, location and bag weight.

The solution

EyeLevel.ai conducted a three-month test using its ChatGPT-like virtual assistant, ingesting PDF documents about KLM and Air France services into its proprietary retrieval-augmented generation (RAG) platform to create the assistant's knowledge base. Air France/KLM experts created questions, reviewed responses and scored accuracy by how closely the bot's answers matched the source documents, then guided an optimization process that customized the bot's responses to the airline's style, formatting and tone.

Reported business value

The assistant reached a 96.2% accuracy rate in the three-month test against a modest 60% target for the first proof of concept. According to the source, the customer service team concluded that EyeLevel.ai dramatically outperformed their expectations and the other solutions they evaluated.