Authors: Anastasiia Barchukova, Antonio Castelli, Artem Alimov, Kate Rasskazimova, Niels Ranzijn, Susanna Agababyan

Every property on Booking.com is listed by a partner — the person or business that owns or manages the place to stay. Some partners are self-managed: they sign up, list their property, and run it entirely on their own, without a dedicated account manager. For them, the registration process — a multi-step flow where they create an account and describe their property — is often where they get stuck, due to its length, unfamiliar terminology, and business-critical decisions they must make before they’ve learned how the platform works.
To help self-managed partners get through registration on their own, we built a chatbot that answers their questions as they come up, without needing a human account manager. Our chatbot should therefore be deeply grounded in the context of the registration funnel, while also being capable of managing partner expectations by answering broader questions about the platform and what comes after registration.
That is the problem this article is about. It describes the approach behind the next iteration of the solution: an agent embedded directly in the registration flow. By design, the assistant is scoped to answering partners’ questions in context, grounded strictly in the partner help center content — it does not generate free-form responses outside that knowledge base.
The system at a glance
At a high level, the chatbot sits inside the registration flow, routes requests through a dedicated backend layer, retrieves relevant help-center content, and uses the conversational AI platform to generate grounded responses. Figure 1 illustrates this simplified end-to-end architecture.

Diagnosing before touching anything
Already in its first iteration, the new agent was using retrieval-augmented generation: it retrieved relevant help-center content and used it to ground its answers. Retrieval itself was working as intended — the right documents were being surfaced for a given query. Yet the answers didn’t fulfill the user’s request.To address this, task completion will be used as the primary metric for evaluating the agent’s quality.
Instead of reaching for the prompt, the team stepped back and looked at the pipeline as a whole — because when a grounded system gives weak answers, the cause is often upstream of the model. That decision is what pointed the investigation at the content the system was retrieving, rather than at the generation step.
The heart of the work: auditing the knowledge base
Early on, the team chose a deliberately simple approach: group the partner help-center articles by intent and pass their content directly to the agent as context, since there was no centralized infrastructure for this internally yet (e.g., an embedding store or semantic search), and the knowledge base was still small. That simplification got the chatbot launched. But it quietly moved the bottleneck somewhere less obvious than the model: the quality and structure of the content being retrieved.
When we actually audited the knowledge base — first running metrics like context relevance, then manually querying the embedding store to see what it returned — rather than the prompts, three structural problems stood out — and none of them were things a better prompt could fix:
- Breadcrumb-like noise in the index. Navigation text and repeated headers, menus, and other page elements were being indexed alongside real answers, making it harder for the system to focus on the useful content.
- Chunking that hid the real context. The articles were split into chunks so the system could build more precise embeddings and retrieve narrower matches. But when a chunk was too small, the retrieved text often lacked the surrounding context needed to answer correctly.
- Irrelevant and risky content. Some embeddings included irrelevant partner-specific content. That information was useless for answering new partners’ questions and risked feeding the chatbot the wrong context.
After these issues were identified, the underlying data was cleaned and the retrieval setup was improved. Rather than a one-off manual cleanup, these fixes were built into the ingestion script that periodically pulls new articles and updates the embedding store: it now filters out noisy or irrelevant content automatically, uses a refined chunking configuration, and expands the retrieved context when needed — so these improvements apply automatically to every future update as new content is added, without manual intervention.
Figure 2 shows the shift: from fixed, intent-based help content to a retrieval flow that assembles fuller context and feeds learnings back into the shared layer.

For broader context on the vector-search infrastructure that supports retrieval-based systems, see How we selected the next vector database at Booking.com.
Expanding retrieval: from one source to several
Cleaning the content was necessary, but the work surfaced a second, distinct issue: a single retrieval source over the general help center simply couldn’t cover everything partners asked during registration. Some questions were about the mechanics of the registration process itself; others touched on legally sensitive topics where a free-form generated answer isn’t acceptable at all. Neither was well served by retrieving from general help articles alone.
So rather than push everything through one index, we expanded the agent to retrieve from purpose-specific sources, each with its own tool and its own contract:
- A registration-process retriever. This tool pulls the most relevant excerpts from a dedicated registration-process knowledge base and returns them as a context string for grounding. It’s designed around intent: the query should carry as many keywords from the partner’s actual question as possible, fall back to the raw user question when the intent is hard to phrase, and rephrase-and-retry when a query comes back empty. That last behavior matters — retrieval is not one-shot, and building the retry into the tool’s contract makes the whole flow more robust to awkwardly worded questions.
- A templated-answers retriever. For legally restricted questions, the correct response isn’t a generated one — it’s a specific, pre-approved static answer we’re allowed to give. This tool matches the partner’s query against that set of templated answers and returns the best fit. Here retrieval isn’t about assembling fuller context at all; it’s a safety and compliance mechanism that keeps the bot from improvising on topics where improvisation is exactly what you don’t want.
The important design point is that these are separate retrievers, not one blended knowledge base. Each source has a different purpose, a different risk profile, and a different notion of what a “good” match means — so keeping them distinct lets the agent reason about which kind of answer a question deserves before it retrieves, instead of hoping a single similarity search happens to surface the right thing.
Re-architecting an agent
With the content cleaned and retrieval expanded across these purpose-specific sources, the system moved to a more explicit agent setup — one that could choose among general help articles, the registration-process retriever, and the templated-answers retriever depending on what the question needed.
This work also reinforced an important lesson: before attributing a failure to the model, it is essential to verify that every part of the retrieval pipeline is working as expected. In this case, an issue that first looked like a model problem turned out to be an upstream data-quality issue.
Shipping it: the online experiment
The current retrieval-focused iteration was released as an online experiment centered on registration-funnel business metrics, so the team could see the change in the context that actually matters: whether partners get further through registration.
What the numbers show
Booking.com runs on experimentation — almost every change ships as an A/B test — so the new version of the chatbot was rolled out as an online experiment alongside the previous one. The scores below come from real partner conversations captured while that experiment was running, scored with Booking.com’s internal evaluation tooling across roughly two thousand dialogues. The evaluation approach behind these metrics is discussed in LLM Evaluation: Practical Tips at Booking.com. Because both versions were served to comparable partner traffic at the same time, the differences reflect the system change rather than seasonal or audience effects.
To understand whether the changes helped partners, we evaluated roughly 2,000 real conversations and looked at both business outcomes and conversation quality.

Task completion was chosen as a main success metric. Under this judge’s definition, a task is completed when the chatbot handles the request directly and sufficiently within its scope and guardrails. For registration and account issues, one grounded troubleshooting step or a necessary clarification question can count as completed; full resolution and user confirmation are not required. The result therefore measures judge-assessed first-line task handling, not confirmed end-to-end resolution.
In a separate evaluation, the score increased 1.33x, suggesting that the chatbot handled substantially more requests directly and sufficiently within its scope. That is an important shift in the role of the assistant: more partners could get a useful first-line answer at the moment they were completing registration, without the conversation immediately stalling. One data caveat is that the required registration-status input was not logged for about 15% of old chatbot conversations, so they were excluded from the primary comparison. The new agent had this field for all conversations. As an extreme conservative sensitivity check, we assigned every missing conversation a perfect score. Chatbot score would then increase, but the updated agent would still lead by a clear margin.
The positive direction therefore remains even under the assumption most favorable to base chatbot. This is a strong positive evidence for improved first-line handling, while using the holdout.
Answer relevance and refusal rate signals show the clearest gains: partners were more likely to receive an answer that addressed their question, and much less likely to encounter a refusal or fallback. In practical terms, the chatbot was not simply responding more often-it was more likely to respond to the question the partner had actually asked, instead of ending the conversation with a dead end.
The durable improvement is underneath these numbers. Cleaning the knowledge base gave retrieval better material to work with. Returning fuller context helped the chatbot connect a partner’s question with the surrounding steps and conditions needed for a useful answer. Removing irrelevant or partner-specific material reduced the risk of grounding an answer in information that did not belong to the person asking.
What we learned
Every meaningful gain in this project came from the retrieval and evaluation layers — the knowledge base, the agent’s choice of when and where to retrieve, and how quality was scored. Four points from that are worth carrying into the next RAG system:
- A business experiment and a quality evaluation answer different questions, so run both. An A/B test tells you whether a metric moved; scoring real conversations from that same experiment tells you what changed underneath it — and knowing where you did not move (accuracy already at ceiling, for instance) is as useful as the wins, because it stops you attributing a gain to the wrong cause.
- Judge an improvement across the whole set of metrics, not the one you set out to move. We were targeting retrieval, yet context relevance moved the least; what showed the new version was clearly better was that answer relevance, refusal rate, and response reliability all improved together. When you change one part of an agent, its value rarely lands in the metric closest to that part — it shows up spread across the others, which is why a single number is never enough to judge the change.
- An automated judge tells you a score is low; it rarely tells you why. We only understood what was wrong with retrieval after reading the retrieved embeddings by hand — that is when the navigation noise, the over-small chunks, and the content that never belonged in the index became obvious. An LLM-as-a-judge can flag that a retrieval is weak, but it won’t tell you a chunk was truncated or the wrong document was indexed. Turning a low score into a concrete fix still took human inspection of what the system actually retrieved.
From Static Help Content to a RAG-Based Onboarding Agent for Partners was originally published in Booking.com ML & DS Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.





