Conditions

TL;DR: You know what your bot should do, but not exactly what it will say. Conditions let you verify the desired outcome — the right data was extracted, the correct flow was triggered, the response had the right intent — without relying on exact wording.

When a bot responds, its output depends on the LLM, the conversation context, external APIs, and how the user phrased their input. You cannot — and should not — assert the exact response text in most cases. What you can assert is whether the bot reached the right state.

Conditions are those assertions. Each condition is attached to a turn and evaluated after the bot responds. If any condition on a turn fails, the turn fails. Unless retries are configured, this also fails the entire test case.

Condition Types

Utterance Condition

Validates the text of the bot's response. This covers everything from checking that a specific phrase is present to evaluating whether the tone is appropriate.

Match types:

Match Type	How it works
Semantic (LLM)	An LLM judges whether the response satisfies a natural language criterion
Exact	The response must match the expected string exactly
Contains	The response must include the expected substring
Regex	The response must match a regular expression pattern

The Semantic match type is particularly useful for subjective qualities — tone, empathy, clarity — that cannot be expressed with a simple string match. You provide an evaluation criterion in natural language (e.g. "Did the bot acknowledge the user's frustration?") and the LLM returns a pass/fail decision with reasoning.

---

Slot Condition

Validates that an extraction field (slot) contains the expected value after the turn. This is useful for verifying that your bot correctly captures information from the conversation — for example, that a user's city was extracted and stored with the right value.

---

LLM Judge Condition

Uses an AI judge to evaluate conversation quality against custom criteria. Unlike the Semantic utterance match, an LLM Judge Condition always references a saved LLM Judge — a reusable Global Agent that encapsulates the LLM configuration, default evaluation instructions, and custom variables.

When you add an LLM Judge Condition to a turn, you select which LLM Judge to use. You can optionally override the default instructions for that specific test case without changing the underlying LLM Judge. This makes it easy to reuse the same judge across many tests while still customizing the evaluation per test.

The judge receives the conversation history and the bot's responses, then returns a pass/fail decision along with its reasoning.

Good LLM Judge instructions are specific and measurable. Compare:

• Vague: "Was the response good?"
• Clear: "Did the bot acknowledge the user's complaint and offer at least one specific next step?"

See Global Agents below for how to create and manage LLM Judges.

---

Tool Call Condition

Verifies that a specific tool or function was called by the bot during the turn. This is the primary way to validate orchestrator routing — confirming that the bot invoked the correct flow or action in response to the user's input.

You specify the exact tool name to check for. Optionally, you can also validate the arguments that were passed to the tool, by providing the expected argument values as a JSON object.

---

Custom Script Condition

Executes a Python script to validate anything that the other condition types cannot express. The script has access to the full conversation state, all events emitted during the turn, the bot's utterances, and any custom variables you define.

Custom Script Conditions are always defined as reusable Global Agents — they cannot be written inline in a test case. This encourages reusability and keeps complex validation logic in one place.

Available context in scripts:

conversation_state.slots       # All extraction field values
conversation_state.latest_message  # The user's input for this turn
bot_utterances                 # List of bot responses
events                         # All events (tool calls, state changes, etc.)
variables                      # Custom variables defined on the condition
llm_api                        # LLM API, if configured on the condition

Return values:

return True                    # Condition passed
return False                   # Condition failed
return True, "message"         # Passed, with a custom message
return False, "reason"         # Failed, with an explanation

Use cases include validating RAG retrieval (checking which knowledge chunks were accessed), inspecting tool call arguments in detail, enforcing business rules that span multiple slots, or calling an external system for comparison.

See Global Agents below for how to create and manage Custom Script Conditions.

---

Global Agents

Global Agents are reusable testing components that are defined once and can be referenced across multiple test cases. There are two types relevant to conditions: LLM Judges, and Custom Script Conditions.

The key advantage of Global Agents is maintainability: when you update an LLM Judge or Custom Script, the change applies to every test case that references it. This is important for shared quality criteria like brand voice or empathy checks, where you want consistent evaluation logic across all tests.

LLM Judges

An LLM Judge defines:

• Evaluation instructions — the default criteria the judge applies (can be overridden per test)
• LLM configuration — which provider to use, temperature, and how much conversation history to include
• Custom variables — parameterized values that can be referenced in the instructions using Jinja2 syntax and overridden per test

LLM Judges are a good fit for quality criteria that appear in many tests: tone of voice, empathy, helpfulness, brand compliance.

Custom Script Conditions

A Custom Script Condition defines:

• A Python script with the validation logic
• Custom variables — parameterized values accessible in the script via variables.get(...) that can be overridden per test
• Optional LLM configuration — if the script needs to call an LLM

Using variables instead of hardcoded values makes Custom Script Conditions reusable across different contexts. For example, a condition that checks which RAG chunks were retrieved can accept the expected chunk names as a variable, so the same script works across many different test cases.

---

Best Practices

Use the right condition type for the job. Slot conditions are the most reliable way to validate extracted data. Tool Call conditions are the right tool for routing assertions. Reserve LLM Judge and Custom Script conditions for logic that genuinely cannot be expressed otherwise — they are more expensive and less deterministic.

Write multiple focused conditions per turn. A single condition that tries to check everything is hard to debug when it fails. Separate concerns: one condition for routing, one for extraction, one for response quality.

Keep LLM Judge instructions specific. Vague criteria lead to inconsistent results. The more concrete and measurable your criterion, the more reliable the judge will be.

Use variables in Global Agents to keep them reusable. Avoid hardcoding values in scripts or instructions. A condition that uses variables can be shared across test cases that differ only in the specific values being checked.