The moment your voice agent actually does something useful (whether that's looking up an order, checking on some inventory, or adding items to a shopping cart) your latency budget gets absolutely screwed.

Here's what a single tool call really ends up costing you:

That's an extra 600-1,200ms of tool calling latency just tacked on top of the original response time, and it gets even worse because these things tend to compound without great memory compression etc.

A reorder workflow where the agent needs to check order history, look up stock levels, create a shopping cart, and then add some items can end up being 4 or 5 sequential tool calls.

The problem of connection overhead

Our first go at this was pretty rough, we opened a fresh connection to the MCP server for every single tool call.

And that added a whopping 500-800ms of overhead before we'd even started doing any real work.

The solution? Open the connection once when the voice session kicks off and then just hold onto it for all the tool calls.

We set it up at the same time as we're doing the rest of the session setup,waiting for the participant, creating the tracking session, and opening the MCP connection all happen concurrently:

# agent.py — setting up concurrently
participant, session_id, mcp = await asyncio.gather(
    ctx.wait_for_participant(),
    _init_session(),
    _init_mcp(),       # MCPConnection().connect() with fallback to one-shot
)

# agent.py — setting up concurrently
participant, session_id, mcp = await asyncio.gather(
    ctx.wait_for_participant(),
    _init_session(),
    _init_mcp(),       # MCPConnection().connect() with fallback to one-shot
)

# agent.py — setting up concurrently
participant, session_id, mcp = await asyncio.gather(
    ctx.wait_for_participant(),
    _init_session(),
    _init_mcp(),       # MCPConnection().connect() with fallback to one-shot
)

500-800ms per tool call → 100-400ms. On a conversation with 5 tool calls, that's 2/ 3 seconds saved.

The problem of context bloat

But connections aren't the only issue. The responses from the tools are also a latency multiplier. Our e-commerce MCP server just returns the full product object: a massive 20+ fields per variant including all sorts of internal IDs, timestamps, SEO metadata, image URLs, tax info. A single list_products call dumps 3-5KB of JSON into the LLM context.

And after a few tool calls we're starting to see the LLM chew through 10,000+ tokens of raw JSON on every single subsequent turn. So the LLM TTFT goes from being around 500ms to being 3+ seconds.

The fix: trim the responses on the server side before we even send them to the agent. Each tool declares a responseShape, a declarative allowlist of fields we actually want to keep:

// modelguide-api — response-trimmer.ts
export type ResponseShape = { [key: string]: true | ResponseShape };

export function trimToShape(
  obj: Record<string, unknown> | null | undefined,
  shape: ResponseShape,
): Record<string, unknown> | null {
  if (obj == null) return null;
  const result: Record<string, unknown> = {};
  for (const [key, spec] of Object.entries(shape)) {
    if (!(key in obj)) continue;
    const value = obj[key];
    if (spec === true) {
      result[key] = value;
    } else if (Array.isArray(value)) {
      result[key] = value.map((item) =>
        item != null && typeof item === "object"
          ? trimToShape(item as Record<string, unknown>, spec)
          : item,
      );
    } else if (value != null && typeof value === "object") {
      result[key] = trimToShape(value as Record<string, unknown>, spec);
    }
  }
  return result;
}

// modelguide-api — response-trimmer.ts
export type ResponseShape = { [key: string]: true | ResponseShape };

export function trimToShape(
  obj: Record<string, unknown> | null | undefined,
  shape: ResponseShape,
): Record<string, unknown> | null {
  if (obj == null) return null;
  const result: Record<string, unknown> = {};
  for (const [key, spec] of Object.entries(shape)) {
    if (!(key in obj)) continue;
    const value = obj[key];
    if (spec === true) {
      result[key] = value;
    } else if (Array.isArray(value)) {
      result[key] = value.map((item) =>
        item != null && typeof item === "object"
          ? trimToShape(item as Record<string, unknown>, spec)
          : item,
      );
    } else if (value != null && typeof value === "object") {
      result[key] = trimToShape(value as Record<string, unknown>, spec);
    }
  }
  return result;
}

// modelguide-api — response-trimmer.ts
export type ResponseShape = { [key: string]: true | ResponseShape };

export function trimToShape(
  obj: Record<string, unknown> | null | undefined,
  shape: ResponseShape,
): Record<string, unknown> | null {
  if (obj == null) return null;
  const result: Record<string, unknown> = {};
  for (const [key, spec] of Object.entries(shape)) {
    if (!(key in obj)) continue;
    const value = obj[key];
    if (spec === true) {
      result[key] = value;
    } else if (Array.isArray(value)) {
      result[key] = value.map((item) =>
        item != null && typeof item === "object"
          ? trimToShape(item as Record<string, unknown>, spec)
          : item,
      );
    } else if (value != null && typeof value === "object") {
      result[key] = trimToShape(value as Record<string, unknown>, spec);
    }
  }
  return result;
}

Result? The LLM sees a stripped back version of the full eCommerce object.

66% token reduction. TTFT dropped from 3s+ down to 0.6-1.8s.

The key decision: trim server-side, not agent-side. Having the shape defined per tool in the MCP server means every connected agent gets the trimmed responses automatically == you don't have to repeat the logic in every voice agent, email agent, or chat agent.

The wrong tool problem

Even which tools are available affects latency. More tools in the LLM context = more tokens per turn = slower TTFT. But the bigger issue is accuracy: with all 11 tools available at once, the LLM makes wrong selections.

Concrete example: during a reorder flow, the customer says "I need more of the Gris tiles." The LLM has product IDs from the order history it just retrieved. But it also has list_products available, so it searches the catalog by name instead of using get_product with the exact ID.

Wrong tool, slower, and sometimes returns the wrong product.

The fix is a guardrail that activates based on conversation state. And the context that arms it; when order history comes back, capture the product IDs.

This is a useful pattern to build on: scoping the available toolset to the current workflow stage reduces the number of tools, which in turn reduces the number of tokens and results in faster time to first token and more accurate tool selection- a double win for performance and accuracy.

Masking tool calling latency with fill lines

Even with persistent connections, trimmed responses, and all the other tricks in the book, tool calls still add anywhere from 600ms to 2 seconds. You can't get rid of that, but you can hide it.

The LLM says a short, specific fill line before calling the tool. LiveKit's AgentSession streams the TTS for that fill line while the tool executes in the background - no custom code needed, the framework handles the overlap naturally.

We enforce this entirely through the system prompt:

# From prompts/base.py
When you call a tool, just say one quick line so the customer
doesn't hear dead air. Make it specific to what you're doing.
Then wait for the result.

Good: "Let me pull up your recent orders."
Good: "Checking stock on those now."
BAD:  "Please hold while I process your request."

# From prompts/base.py
When you call a tool, just say one quick line so the customer
doesn't hear dead air. Make it specific to what you're doing.
Then wait for the result.

Good: "Let me pull up your recent orders."
Good: "Checking stock on those now."
BAD:  "Please hold while I process your request."

# From prompts/base.py
When you call a tool, just say one quick line so the customer
doesn't hear dead air. Make it specific to what you're doing.
Then wait for the result.

Good: "Let me pull up your recent orders."
Good: "Checking stock on those now."
BAD:  "Please hold while I process your request."

Each tool has its own specific fill line prescribed in the prompt.

No parallel dispatch at code level.

The fill line pattern is entirely prompt-enforced, and it works because the framework is already streaming TTS before the tool result comes back.