Skip to main content

Input Generation

By default, each turn in a test case uses a fixed user input — a specific message that is sent to the bot every time the test runs. Input Generation is an alternative mode where an LLM generates the user message at runtime based on natural language instructions.

Why Use Generated Inputs​

Fixed inputs are simple and fully reproducible, which makes them a good default. But they have a limitation: they test exactly one phrasing. If your bot handles the same topic slightly differently depending on how the user phrases it, a fixed input only covers one case.

Generated inputs address this by simulating natural variation. Instead of specifying the exact message, you describe what the user should say — and the LLM generates a realistic phrasing each time the test runs. This makes tests less brittle and better at catching regressions in how the bot understands and responds to users.

Generated inputs work well when:

  • You want to verify that the bot responds correctly across different phrasings
  • You are testing bot behavior that should be robust to natural language variation
  • You want to simulate different user personas or communication styles

Fixed inputs remain the better choice when:

  • You need 100% reproducible test runs (e.g. for debugging a specific transcript)
  • You are testing exact phrase handling or specific commands
  • You want to minimize test execution cost and latency

How It Works​

When a turn is configured to use input generation, the system calls an Input Generator (a Global Agent — see below) to produce the user message before sending it to the bot. You provide generation instructions that describe what the user should say, such as:

"Ask about the delivery status of a recent order"

The Input Generator uses those instructions, the conversation history up to that point, and its configured LLM to generate a realistic user message. The generated message is then used as the actual input for that turn.

Retries​

When using generated inputs, you can configure a maximum number of retries for the turn. If the generated input causes the turn's conditions to fail, the system regenerates a new input and retries — up to the configured limit. This is useful when occasional mismatches are expected due to phrasing variation, and you want the test to pass as long as at least one attempt succeeds.

Importantly, a retry does not restart the conversation from scratch. Each retried input is sent as a continuation of the existing conversation, preserving all prior turns and context. This means the bot sees the full history, including the previous (failed) attempt's exchange, when processing the retried input.

Each attempt is recorded in the execution results, so you can inspect which phrasing was generated and how the bot responded.

Global Agents: Input Generators​

An Input Generator is a reusable Global Agent that controls how user inputs are generated. It defines:

  • LLM configuration — which provider and model to use, temperature, and how much conversation history to include as context
  • Generation prompt — the system prompt template that instructs the LLM how to produce inputs, written in Jinja2; has access to {{ user_provided_instructions }}, {{ conversation_history }}, and any custom variables
  • Default generation instructions — optional default instructions that pre-fill the per-turn instructions field; can be overridden per turn
  • Custom variables — parameterized values that can be referenced in the generation prompt and overridden per turn
  • Agent — the Python agent that processes the generation request; the built-in default agent handles most scenarios, but a custom Python agent can be used for advanced control

Every bot comes with a Default Input Generator that works out of the box with sensible settings. For most use cases, you only need to provide the generation instructions per turn and leave the generator on its default.

Custom Variables for Reusability​

Custom variables on an Input Generator allow a single generator to be reused across different test scenarios. For example, a "Persona Simulator" generator might define a persona variable:

Default: "curious customer"

Different turns can then override persona to simulate different user types:

  • "frustrated customer" — for testing empathy handling
  • "technical user" — for testing advanced support flows
  • "new user unfamiliar with the product" — for testing onboarding

This avoids creating a separate generator for each persona.

Custom Agents​

For advanced scenarios, the Python agent that handles generation can be replaced with a custom implementation. A custom agent receives the user instructions, conversation history, and variables, and can apply arbitrary logic to produce the generated input — for example, to enforce domain-specific constraints or integrate with external data sources.

Custom agents are defined in the Scripts section of your bot and selected on the Input Generator configuration.

Best Practices​

Write clear generation instructions. The more specific your instructions, the more reliably the LLM generates relevant inputs. Describe intent and context, not exact wording.

Use lower temperature for more consistency. A temperature of 0.5–0.7 gives good variation while staying on-topic. Very high temperatures may produce off-topic or unrealistic inputs.

Pair with behavioral conditions. Since the exact phrasing is unpredictable, avoid Exact or Contains utterance conditions when using generated inputs. Instead, rely on Tool Call conditions (to verify routing), Slot conditions (to verify extraction), and LLM Judge conditions (to evaluate response quality).

Use fixed inputs for debugging. When a test is failing and you need to reproduce it consistently, switch to a fixed input to isolate the problem. Generated inputs make debugging harder because each run is different.