The harness includes first-class evaluation capabilities powered by the LangSmith evaluation framework (evaluate / aevaluate). You can benchmark tool selection, skill routing, multi-agent delegation, and guardrails either locally without uploading results or in the LangSmith Web UI. Local evaluation still invokes the supplied agent and may incur model/tool costs. The scripted uv run python -m examples.run_evals --offline example needs no model API key.

Enable tracing

Install the LangSmith extra and configure LANGSMITH_API_KEY and your project. The extension records agent turns, model calls, tool calls, and supported nested delegation spans. When project_name is omitted, LangSmithExtension uses the harnessx-agents project.

from harnessx import Agent, LangSmithExtension

agent = Agent(extensions=[
    LangSmithExtension(project_name="agent-development", tags=["development"]),
])

Tracing can transmit prompts and tool payloads to LangSmith; enable it explicitly for the data you intend to send. Trace delivery is best effort and is not an execution guard. Use the flight recorder when you need captured runtime boundaries for offline inspection.

Running an evaluation

from harnessx import Agent, AgentConfig, PermissionLevel
from examples._calculator import calculate  # Shared helper when running from a checkout.
from harnessx.evals import (
    build_example,
    default_evaluators,
    evaluate_agent,
    tool_selection_evaluator,
    contains_evaluator,
)

# 1. Define your agent factory or instance
def make_agent(inputs):
    agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))
    agent.tools.register_tool(calculate, permission=PermissionLevel.ALLOW, replay_policy="safe")
    return agent

# 2. Define test cases or use built-in suites ("tool_calling", "skills", "all")
dataset = [
    build_example(
        inputs={"prompt": "What is 144 / 12?"},
        outputs={
            "expected_tools": ["calculate"],
            "contains_all": ["12"],
            "max_allowed_iterations": 3,
        },
    )
]

# 3. Evaluate against the live model, keeping results local.
summary = evaluate_agent(
    agent=make_agent,
    dataset=dataset,
    evaluators=[tool_selection_evaluator, contains_evaluator],
    experiment_prefix="math-agent-benchmark",
    offline=True,   # Disables LangSmith upload; the agent still calls its model.
)

print(f"Pass rate: {summary.pass_rate * 100:.1f}%")

Experiments default to the prefix harnessx-eval; built-in datasets synced to LangSmith are prefixed harnessx-.

Evaluate from a running event loop

evaluate_agent_async(...) awaits the same evaluation from async code. It runs the synchronous runner in a worker thread, so pass a factory rather than an Agent bound to your loop. It cannot be cancelled once started and must not be nested inside another evaluation.

from harnessx.evals import evaluate_agent_async

async def benchmark():
    summary = await evaluate_agent_async(
        agent=make_agent,      # a factory, not an Agent bound to this loop
        dataset="tool_calling",
        offline=True,
    )
    return summary.pass_rate

Running evals from the CLI

# Scripted integration demonstration (langsmith extra, no API keys required)
uv run python -m examples.run_evals --offline

# Run live model benchmarks without upload (model API key required)
python -m harnessx.evals.cli --suite tool_calling --offline

# Run against Anthropic and upload results + traces to LangSmith
export LANGSMITH_API_KEY="lsv2_pt_..."
python -m harnessx.evals.cli --suite skills --model claude-sonnet-4-6 --upload

# Run all benchmark suites with concurrency
python -m harnessx.evals.cli --suite all --concurrency 2

When uploaded, the CLI prints a clickable LangSmith URL to inspect row-level scores, side-by-side prompt diffs, and the complete nested execution tree for every turn.