toolsnap is a zero-dependency, SDK-agnostic recorder and replayer for LLM agent tool calls. The idea is simple: record the real trajectory once, then replay and assert on it forever. This post walks through the motivation, the design choices, and how to use it.

Motivation

LLM agents are non-deterministic at the model level — ask the same question twice and you may get a different answer. But the agent's tool-call trajectory — which tools it called, in what order, with what arguments — is the real observable behavior that matters for correctness. If a prompt change silently makes your agent call summarize before search, or drop a tool call entirely, that is a behavioral regression worth catching. Yet most testing approaches make this surprisingly hard to do reliably.

The Problem with Testing LLM Agents

There are three dominant approaches to testing agents, and each has a meaningful shortcoming.

Live APIs in Every Test Run

Run the real agent against the real APIs in CI. This is the most honest approach but has real costs:

• Slow and expensive — tool backends (search APIs, databases, external services) are called on every test run.
• Requires credentials in CI — a security and operational headache.
• Non-deterministic — the LLM may call tools with different arguments each run, so a test that passed yesterday can fail today with no code change.

Hand-Written Mocks

Patch the tool with a hardcoded return value: mock.return_value = ["doc1", "doc2"]. The fundamental problem is that you wrote that value yourself — before you knew what the real agent produces. If the agent never calls the tool, the mock never fails. You are testing a fiction, not real behavior.

Network-Level Recording

Tools like vcrpy or pytest-recording record all HTTP traffic. This captures LLM requests alongside tool requests, which means fixtures balloon in size and become fragile to any SDK update, header change, or streaming format shift. They also don't operate at the clean Python function boundary where the tool actually lives.

The Idea: Record the Trajectory

toolsnap takes a different approach. Instead of mocking or hitting live APIs in CI, it records real tool calls at the Python function boundary during a one-time live run, then replays those exact responses in every subsequent test run.

Here is what a recording run looks like internally:

# Live run:  agent.run("find llm docs")
#   └─ search(query="llm agents")    → real API → ["doc1", "doc2"]  ← saved
#   └─ summarize(text="doc1 | doc2") → real API → "LLMs use tools" ← saved

# Test run:  agent.run("find llm docs")
#   └─ search(query="llm agents")    → toolsnap returns ["doc1", "doc2"]
#   └─ summarize(text="doc1 | doc2") → toolsnap returns "LLMs use tools"
#
#   No tool backends called. Same responses. Same agent decisions. Same trajectory.

Each recorded call stores: function name, arguments, return value, wall-clock duration, and any raised exception. The fixture is a plain JSONL file — one line per call — readable, diffable, and version-controllable.

toolsnap

toolsnap offers three progressively richer ways to use it, depending on how much control you need.

@snap and @replay

The simplest entry point. @snap wraps a function and records every call to a JSONL fixture. @replay wraps the same function signature and replays the recorded responses — you don't even need to provide a body.

# main.py — run once against live APIs to create the fixture
from toolsnap import snap

@snap                         # auto-saves to fixtures/search.jsonl
def search(query: str) -> list[str]:
    return real_search_api(query)

@snap("fixtures/weather.jsonl")   # explicit path
def get_weather(city: str) -> dict:
    return real_weather_api(city)

agent.run("what's the weather in london and find llm docs")
# fixtures/search.jsonl and fixtures/weather.jsonl written

# test_agent.py — tool backends don't run; the LLM still does
from toolsnap import replay

@replay                            # reads from fixtures/search.jsonl
def search(query: str) -> list[str]: ...

@replay("fixtures/weather.jsonl")
def get_weather(city: str) -> dict: ...

def test_agent_trajectory():
    result = agent.run("what's the weather in london and find llm docs")
    # search() and get_weather() returned their recorded responses
    # assert on the agent's output or behaviour here
    assert result is not None

The fixture path can be omitted (defaults to fixtures/{fn_name}.jsonl), an explicit file path, or a directory. The recording uses call-index ordering, so replays are deterministic even when arguments vary slightly between runs.

SnapSession

When your agent calls multiple tools in a single run, SnapSession wraps them all under one fixture and provides the full assertion API. This is the recommended approach for multi-step agents.

from toolsnap import SnapSession, contains

# Recording
with SnapSession.snap("fixtures/session.jsonl") as s:
    s.wrap(search)
    s.wrap(summarize)
    agent.run("find and summarize llm docs")
# All calls to search() and summarize() saved to fixtures/session.jsonl

# Replaying and asserting
with SnapSession.replay("fixtures/session.jsonl") as s:
    s.wrap(search)
    s.wrap(summarize)
    agent.run("find and summarize llm docs")

s.assert_called("search", times=1)
s.assert_called_with("search", query=contains("llm"))
s.assert_call_order(["search", "summarize"])
s.assert_no_errors()

The full set of assertion methods available on a session:

In strict mode (the default), any tool call that exceeds the recorded fixture raises UnexpectedToolCall. This catches the case where a prompt change causes the agent to call tools it previously didn't — trajectory drift equals behavioral drift.

pytest Plugin

For teams, the pytest plugin is the cleanest setup. Record and replay modes are controlled from the command line — no code changes needed between recording and testing.

# conftest.py
pytest_plugins = ["toolsnap.pytest_plugin"]

# test_agent.py
@pytest.mark.toolsnap_fixture("fixtures/session.jsonl")
def test_agent_trajectory(toolsnap_session):
    toolsnap_session.wrap(search)
    toolsnap_session.wrap(summarize)
    agent.run("find and summarize llm docs")

    toolsnap_session.assert_called("search", times=1)
    toolsnap_session.assert_call_order(["search", "summarize"])

pytest tests/                      # replay — tool backends free, LLM still runs
pytest tests/ --toolsnap-record    # re-record after prompt/code changes
pytest tests/ --toolsnap-strict=false  # allow unexpected calls to fall through

The plugin automatically warns when a fixture may be stale — for instance, when a tool's function signature has changed since the recording was made.

Assertion Predicates

All assertion methods accept predicate objects for structural matching instead of exact values. This is useful when you care about the shape of an argument rather than its exact string.

from toolsnap import contains, matches, any_of, gt, lt

s.assert_called_with("search", query=contains("london"))
s.assert_called_with("get_weather", city=any_of("london", "paris"))
s.assert_called_with("embed", n_tokens=lt(512))
s.assert_called_with("fetch", url=matches(r"https://.*\.json$"))

CLI: Inspect and Diff Fixtures

toolsnap ships a small CLI for working with fixture files directly — useful after a re-record to understand what changed.

toolsnap diff fixtures/before.jsonl fixtures/after.jsonl
# Diff: fixtures/before.jsonl → fixtures/after.jsonl
# ────────────────────────────────────────────────────
#   search      call 0  args unchanged   result CHANGED (3 items → 2 items)
# + summarize   call 0  ADDED

toolsnap show fixtures/session.jsonl     # pretty-print all recorded calls
toolsnap stats fixtures/session.jsonl    # call counts, avg/p95 latency, errors
toolsnap validate fixtures/session.jsonl # check fixture integrity
toolsnap list                            # list all known fixture files

The diff command is particularly useful after a prompt change: it tells you exactly which tool calls changed, which were added, and which were dropped — giving you a precise signal of what shifted in the agent's behavior.

When to Use (and When Not To)

toolsnap is not a general-purpose mock framework or a network recorder. It fits a specific gap.

One important clarification: the LLM still runs in replay mode. toolsnap only freezes the tool backends. This means the test is not purely offline, but it is offline for everything that matters most — the slow, expensive, credential-requiring external services.

Conclusion

Agent behavior is defined by what the agent does, not just what it says. toolsnap makes the tool-call trajectory the first-class test artifact: record once from a real run, replay deterministically in CI, and assert on the exact shape of what happened. When behavior drifts — and it will, with prompt changes, dependency updates, or model upgrades — toolsnap surfaces it immediately.

Source: github.com/waitasecant/toolsnap

PyPI: pypi.org/project/toolsnap/