Skip to main content

Testing Overview

TL;DR: Testing allows you to automatically validate your bot's behavior by creating test cases with turns (user inputs) and conditions (validation rules). Create a test, add turns, define what should happen, run it, and get instant pass/fail results.

Why Testing Matters​

A bot is rarely a self-contained system. It typically depends on third-party LLMs whose behavior can shift between model versions, external APIs that can change or become unavailable, and complex flow logic where a small modification can have unintended side effects elsewhere. Any of these can break a bot's purpose at any moment — not just when you deploy a change, but also when something outside your control changes.

Waiting for a customer to report that the bot stopped working is not a strategy. Neither is manually walking through every conversation path after every change. Automated tests let you define the expected behavior once and verify it continuously, so you catch problems before your users do.

Core Concepts​

Test Case​

A test case models a single conversation scenario. It consists of one or more turns and defines what the bot should do at each step. A test case typically represents one specific user flow — for example, a user requesting an address change, or asking for their order status.

Turn​

A turn is one exchange: the user sends a message, the bot responds. A test case can have multiple turns to simulate a full multi-step conversation. Each turn carries its own user input and a set of conditions that validate the bot's behavior for that step.

Condition​

Bot builders know what they want their bot to do — but they cannot know exactly what it will say. The response depends on the LLM, the conversation history, third-party data, and how the user phrased their input. The exact wording is unpredictable by design.

Conditions bridge this gap. Rather than asserting exact outputs, they verify that the bot reached the desired state: the right data was extracted, the correct flow was triggered, the response carried the right intent. A condition is a validation rule attached to a turn that answers the question: "did the bot do what it was supposed to do?"

After the bot responds, all conditions on that turn are evaluated to determine whether the turn passed or failed. Conditions can check the bot's text response, which tools were called, what data was extracted, or apply AI-based quality evaluation. See Conditions for all available types.

Execution​

When you run a test case, the system simulates the conversation turn by turn. Each turn is evaluated against its conditions. Execution stops at the first failure, so you get a clear signal of exactly where behavior diverged from expectations.

Test Suite​

A test suite is a named collection of test cases. Suites let you group related tests — for example, all tests for your checkout flow — and run them together. See Test Suites for details.

Getting Started​

The fastest way to create a first test case is to have a real conversation with your bot in the Development Chat, then convert it directly into a test case. The system generates all turns from the conversation automatically. From there, you add conditions to define what passing means for each turn.

If you want to see this in practice before diving into the reference pages, the Address Change example walks through a complete test case from start to finish — including two different strategies for how to design it.

What's Next​

  • Conditions — Validate bot responses, extracted data, tool calls, and conversation quality
  • Input Generation — Use LLM-generated user inputs for more realistic and varied tests
  • Test Suites — Group test cases and run them together
  • Test Execution — How test execution works and how to interpret results
  • Example: Address Change — A concrete walkthrough of a real test case