tool_called — Was a specific tool invoked?

Use when: You need to verify the agent used a particular tool at any point during the session.

{
  type: "tool_called";
  toolSlug: string;         // bare slug, e.g. "get_order"
  catalogSlug?: string;     // for multi-connector SOPs
}

{
  type: "tool_called";
  toolSlug: string;         // bare slug, e.g. "get_order"
  catalogSlug?: string;     // for multi-connector SOPs
}

{
  type: "tool_called";
  toolSlug: string;         // bare slug, e.g. "get_order"
  catalogSlug?: string;     // for multi-connector SOPs
}

How it works: Scans session tool call log for at least one call matching the resolved tool name.

Cost: Zero (pure log scan). Use this as the default when you just need to confirm tool usage.

tool_sequence — Were tools called in the right order?

Use when: The procedure requires tools to be called in a specific sequence (e.g., lookup before modify).

{
  type: "tool_sequence";
  sequence: string[];       // ["get_order", "set_delivery_address"]
  contiguous: boolean;      // false = other tools allowed between
  catalogSlug?: string;
}

{
  type: "tool_sequence";
  sequence: string[];       // ["get_order", "set_delivery_address"]
  contiguous: boolean;      // false = other tools allowed between
  catalogSlug?: string;
}

{
  type: "tool_sequence";
  sequence: string[];       // ["get_order", "set_delivery_address"]
  contiguous: boolean;      // false = other tools allowed between
  catalogSlug?: string;
}

How it works: Walks the tool call log checking that tools appear in order. Supports | alternatives: "get_order|look_up_order" matches either.

Cost: Zero (log scan). Prefer this over multiple tool_called steps when order matters.

tool_input_contains — Did the tool receive correct input?

Use when: You need to verify the agent passed specific parameters to a tool (e.g., correct order ID format, required fields present).

{
  type: "tool_input_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, InputAssertion>;
}

type InputAssertion =
  | { op: "exists" }                              // field is present
  | { op: "equals"; value: string | number | boolean }
  | { op: "matches"; pattern: string }            // regex
  | { op: "gt"; value: number }
  | { op: "lt"; value: number };

{
  type: "tool_input_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, InputAssertion>;
}

type InputAssertion =
  | { op: "exists" }                              // field is present
  | { op: "equals"; value: string | number | boolean }
  | { op: "matches"; pattern: string }            // regex
  | { op: "gt"; value: number }
  | { op: "lt"; value: number };

{
  type: "tool_input_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, InputAssertion>;
}

type InputAssertion =
  | { op: "exists" }                              // field is present
  | { op: "equals"; value: string | number | boolean }
  | { op: "matches"; pattern: string }            // regex
  | { op: "gt"; value: number }
  | { op: "lt"; value: number };

Cost: Zero (log scan + assertion check).

tool_output_contains — Did the tool return expected data?

Use when: You need to verify pre-conditions based on tool output (e.g., order status is not "shipped" before allowing modification).

{
  type: "tool_output_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, OutputAssertion>;
}

type OutputAssertion =
  | { op: "exists" }
  | { op: "equals"; value: string | number | boolean }
  | { op: "contains"; value: string }
  | { op: "not_in"; values: string[] };

{
  type: "tool_output_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, OutputAssertion>;
}

type OutputAssertion =
  | { op: "exists" }
  | { op: "equals"; value: string | number | boolean }
  | { op: "contains"; value: string }
  | { op: "not_in"; values: string[] };

{
  type: "tool_output_contains";
  toolSlug: string;
  catalogSlug?: string;
  assertions: Record<string, OutputAssertion>;
}

type OutputAssertion =
  | { op: "exists" }
  | { op: "equals"; value: string | number | boolean }
  | { op: "contains"; value: string }
  | { op: "not_in"; values: string[] };

Cost: Zero (log scan + assertion check).

no_tool_called — Were forbidden tools avoided?

Use when: The agent must NOT call certain tools in this scenario (e.g., no direct refund processing for high-value orders).

{
  type: "no_tool_called";
  toolSlugs: string[];      // ["complete_cart"] — these must NOT appear
  catalogSlug?: string;
}

{
  type: "no_tool_called";
  toolSlugs: string[];      // ["complete_cart"] — these must NOT appear
  catalogSlug?: string;
}

{
  type: "no_tool_called";
  toolSlugs: string[];      // ["complete_cart"] — these must NOT appear
  catalogSlug?: string;
}

Cost: Zero (negative log scan).

confirmation_requested — Did the agent ask before mutating?

Use when: A mutation tool (e.g., set_delivery_address) requires the agent to ask the customer for confirmation first.

{
  type: "confirmation_requested";
  beforeToolSlug: string;   // the mutation tool
  catalogSlug?: string;
  pattern: string;           // regex matching confirmation language
  withinLastNTurns?: number; // how far back to look (default: 4)
}

{
  type: "confirmation_requested";
  beforeToolSlug: string;   // the mutation tool
  catalogSlug?: string;
  pattern: string;           // regex matching confirmation language
  withinLastNTurns?: number; // how far back to look (default: 4)
}

{
  type: "confirmation_requested";
  beforeToolSlug: string;   // the mutation tool
  catalogSlug?: string;
  pattern: string;           // regex matching confirmation language
  withinLastNTurns?: number; // how far back to look (default: 4)
}

How it works: Finds the tool call in the log, then looks at preceding assistant messages (within the turn window) for a regex match against confirmation language.

Cost: Zero (log scan + regex).

llm_judge — LLM-evaluated natural-language criterion

Use when: The check cannot be expressed as a tool log assertion — tone, policy adherence, reasoning quality, behavioral requirements.

{
  type: "llm_judge";
  criterion: string;        // "Agent verifies customer identity before accessing account"
  model?: string;           // override default judge model
  rubric?: string;          // detailed rubric for nuanced evaluation
}

{
  type: "llm_judge";
  criterion: string;        // "Agent verifies customer identity before accessing account"
  model?: string;           // override default judge model
  rubric?: string;          // detailed rubric for nuanced evaluation
}

{
  type: "llm_judge";
  criterion: string;        // "Agent verifies customer identity before accessing account"
  model?: string;           // override default judge model
  rubric?: string;          // detailed rubric for nuanced evaluation
}

How it works: Sends the full conversation transcript + criterion (+ optional rubric) to an LLM judge. Returns pass/fail with reasoning.

Cost: External LLM call (latency + cost). This is the only evaluator that makes external calls. Use deterministic evaluators first; reserve llm_judge for checks that genuinely need language understanding.

Best practices for criterion writing:

• Be specific: "Agent asked for name AND order ID" > "Agent verified identity"

• Include temporal constraints: "BEFORE performing any order lookup" not just "at some point"

• State what should NOT happen when relevant: "Agent did NOT process refund directly"

• If the criterion is complex, use the rubric field for a detailed scoring guide