Skip to main content

Example: Address Change

This page demonstrates two different strategies for testing the same address change flow. They represent different trade-offs between control and flexibility:

  • Approach 1: Explicit turns — Each user input is defined manually, turn by turn. This gives full control over the conversation path and is well-suited for verifying routing and extraction at specific steps.

  • Approach 2: Input generation with retries — A single turn with generated inputs and retries lets the bot drive the conversation to completion. This approach focuses on the end state rather than the path, and tests whether the bot can reach the correct outcome across a natural, open-ended exchange.

Approach 1: Explicit Turns​

The Conversation​

The test is based on a real conversation captured in the Development Chat. A user wants to register a new address in Berlin:

The bot handles this in three turns: it asks for the city, then the zip code, then confirms the registration. This multi-turn flow is exactly the kind of scenario worth automating — it has multiple extraction steps and a specific routing requirement that could silently break after a change.

The fastest way to turn this into a test case is to convert the conversation directly — this is possible both from the Development Chat and from any conversation in the Chat History. The system generates a test case with all three turns pre-populated.

The auto-generated test includes utterance conditions on the first two turns and slot conditions on the third. It is a useful baseline, but it is not complete — it does not verify that the bot actually routed to the correct flow. That is what the conditions below add.

The Conditions​

Turn 1 — Routing​

The most critical thing to verify in Turn 1 is that the bot routes to the correct flow. If the orchestrator misidentifies the intent — routing to a general inquiry flow instead of the address change handler — the address change will never be processed, even if the conversation appears to go well from the outside.

This is not something the auto-generated test covers. A Tool Call Condition on address_change_flow needs to be added manually. It fails if that tool was not invoked during Turn 1.

Turn 2 — Follow-up​

Turn 2 is a natural language exchange where the user provides the city. An Utterance Condition can verify the bot follows up correctly — for example, checking that the response asks for a zip code.

Turn 3 — Extraction and Confirmation​

By Turn 3, the bot should have extracted both values from the conversation. Two Slot Conditions verify this:

  • town == Berlin
  • zip_code == 10155

Even if the conversation flows naturally, incorrect extraction means the address change won't be processed correctly downstream. These conditions catch that class of failure before it reaches production.

An Utterance Condition can additionally verify the bot's confirmation message at the end of the turn.

The Full Validation Chain​

Together, these conditions cover the critical failure modes for this flow:

  • Wrong routing — Tool Call Condition on Turn 1
  • Wrong extraction — Slot Conditions on Turn 3
  • Missing confirmation — Utterance Condition on Turn 3

Any regression in the LLM behavior, the flow logic, or the extraction configuration will surface here before a real user encounters it.

Approach 2: Input Generation with Retries​

This approach uses a single turn with an input generator and retries to test the same flow. Instead of scripting each exchange, you describe the user's goal and let the bot lead the conversation to completion.

The Setup​

The test case has a single turn. The turn uses input generation with the following generation instructions:

"You are a friendly user who wants to change their address. Your new address is 10155, Berlin. Only provide these details one after another if you are asked for them."

The turn is configured to allow up to 10 retries.

Two Slot Conditions are added to this turn:

  • town == Berlin
  • zip_code == 10155

How It Executes​

When the test runs, the input generator produces an opening message — something like "I'd like to update my address." The bot responds and asks for the city. The conditions are evaluated: neither slot is set yet, so the turn fails. Because retries are configured, the turn runs again.

On the next attempt, the input generator has the conversation history as context and produces the city — "Berlin". The bot asks for the zip code. The conditions are evaluated again: town is now set but zip_code is not. The turn retries once more.

On the following attempt, the generator provides the zip code. The bot confirms the registration. Both slot conditions now pass, and the test passes.

When to Use This Approach​

This approach is a good fit when you want to verify that the bot can successfully complete a flow end-to-end, without tying the test to a specific conversation path. It is more resilient to changes in how the bot phrases its questions, and it does not require you to know the exact sequence of exchanges in advance.

The trade-off is that it does not verify routing or intermediate steps. If the bot reaches the right outcome via the wrong flow, this test will not catch it. For that, Approach 1 with an explicit Tool Call Condition is the better choice.

The two approaches are complementary: Approach 1 gives you precision and routing coverage; Approach 2 gives you end-state coverage with less maintenance overhead.