# Tracing & evaluations

Capture behavior and evaluate workflow outcomes.

The harness includes first-class evaluation capabilities powered by the
**LangSmith evaluation framework** (`evaluate` / `aevaluate`). You can benchmark
tool selection, skill routing, multi-agent delegation, and guardrails either
locally without uploading results or in the LangSmith Web UI. Local evaluation
still invokes the supplied agent and may incur model/tool costs. The scripted
`uv run python -m examples.run_evals --offline` example needs no model API key.

## Enable tracing

Install the LangSmith extra and configure LANGSMITH_API_KEY and your project.
The extension records agent turns, model calls, tool calls, and supported nested
delegation spans. When `project_name` is omitted, `LangSmithExtension` uses the
`harnessx-agents` project.

```python
from harnessx import Agent, LangSmithExtension

agent = Agent(extensions=[
    LangSmithExtension(project_name="agent-development", tags=["development"]),
])
```

Tracing can transmit prompts and tool payloads to LangSmith; enable it explicitly
for the data you intend to send. Trace delivery is best effort and is not an
execution guard. Use the flight recorder when you need captured runtime
boundaries for offline inspection.

## Running an evaluation

```python
from harnessx import Agent, AgentConfig, PermissionLevel
from examples._calculator import calculate  # Shared helper when running from a checkout.
from harnessx.evals import (
    build_example,
    default_evaluators,
    evaluate_agent,
    tool_selection_evaluator,
    contains_evaluator,
)

# 1. Define your agent factory or instance
def make_agent(inputs):
    agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))
    agent.tools.register_tool(calculate, permission=PermissionLevel.ALLOW, replay_policy="safe")
    return agent

# 2. Define test cases or use built-in suites ("tool_calling", "skills", "all")
dataset = [
    build_example(
        inputs={"prompt": "What is 144 / 12?"},
        outputs={
            "expected_tools": ["calculate"],
            "contains_all": ["12"],
            "max_allowed_iterations": 3,
        },
    )
]

# 3. Evaluate against the live model, keeping results local.
summary = evaluate_agent(
    agent=make_agent,
    dataset=dataset,
    evaluators=[tool_selection_evaluator, contains_evaluator],
    experiment_prefix="math-agent-benchmark",
    offline=True,   # Disables LangSmith upload; the agent still calls its model.
)

print(f"Pass rate: {summary.pass_rate * 100:.1f}%")
```

Experiments default to the prefix `harnessx-eval`; built-in datasets synced to
LangSmith are prefixed `harnessx-`.

## Evaluate from a running event loop

`evaluate_agent_async(...)` awaits the same evaluation from async code. It runs
the synchronous runner in a worker thread, so pass a factory rather than an
`Agent` bound to your loop. It cannot be cancelled once started and must not be
nested inside another evaluation.

```python
from harnessx.evals import evaluate_agent_async

async def benchmark():
    summary = await evaluate_agent_async(
        agent=make_agent,      # a factory, not an Agent bound to this loop
        dataset="tool_calling",
        offline=True,
    )
    return summary.pass_rate
```

## Running evals from the CLI

```bash
# Scripted integration demonstration (langsmith extra, no API keys required)
uv run python -m examples.run_evals --offline

# Run live model benchmarks without upload (model API key required)
python -m harnessx.evals.cli --suite tool_calling --offline

# Run against Anthropic and upload results + traces to LangSmith
export LANGSMITH_API_KEY="lsv2_pt_..."
python -m harnessx.evals.cli --suite skills --model claude-sonnet-4-6 --upload

# Run all benchmark suites with concurrency
python -m harnessx.evals.cli --suite all --concurrency 2
```

When uploaded, the CLI prints a clickable LangSmith URL to inspect row-level scores,
side-by-side prompt diffs, and the complete nested execution tree for every turn.