# Flight recorder

Export a portable incident and inspect it offline.

The recorder preserves model/tool attempts and their raw/processed boundaries in
the durable runtime. Export a run as a portable `.hx` archive, verify it, and inspect
the ordered timeline without executing models, handlers, bundled code, or services.
It does not introduce symbolic memory or change tool signatures.

## SDK usage

```python
from harnessx import AgentRuntime, SQLiteBackend, ExportPolicy, IncidentRecorder

async with SQLiteBackend("runtime.db") as backend:
    async with AgentRuntime(agent, backend=backend, recording=True) as runtime:
        result = await runtime.run("Analyze overdue invoices")
        bundle = await runtime.export_incident(
            result.run_id,
            destination="incident.hx",  # Existing files are never overwritten.
            policy=ExportPolicy(include_payloads=True, include_artifacts=True),
        )

recorder = IncidentRecorder()
report = await recorder.verify(bundle)
playback = await recorder.playback(bundle)
for record in playback.records:
    print(record["seq"], record["kind"], record["attempt_id"])
```

Recording is off for new sessions unless explicitly enabled. The setting is saved
with the session. On `resume()`, an omitted setting inherits that saved choice;
an explicit incompatible setting is rejected. Recording cannot be enabled
retroactively for an unrecorded run. Ordinary `Agent.run()` has no recorder option.

To export after a crash **without resuming execution**, open the store directly:

```python
from harnessx import SQLiteBackend, export_incident, ExportPolicy

async with SQLiteBackend("runtime.db") as store:  # initialises on entry, closes on exit
    bundle = await export_incident(
        store, session_id, run_id, "crashed-run.hx",
        policy=ExportPolicy(include_payloads=True),
    )
```

The application must authorize access to the session before exposing either export
operation. A run is checked against its supplied/bound session, but knowing those
IDs is not authentication. Opening an older store with automatic migration enabled
can upgrade its schema; export never starts agent execution.

## Recorded boundaries and recovery

Each private journal entry has a UUID, contiguous per-run sequence, format version,
session/run/step/attempt IDs, timestamp, optional execution key, and JSON payload.
It is separate from the public `RunEvent` stream: pre-middleware responses must not
leak into that stream. The journal also retains copies of public events, including
stream deltas that the engine actually emits.

| Boundary | Meaning |
|---|---|
| `run.started`, `run.resumed` | Initial request/configuration and explicit resume history. |
| `model.started` | Provider-neutral request committed before provider invocation. |
| `model.response`, `model.processed` | Normalized response before and after middleware. Provider SDK transport objects are excluded. |
| `model.completed`, `model.failed`, `model.interrupted` | Engine transition, captured failure, or interruption. Retries have distinct attempt IDs. |
| `tool.started`, `tool.dispatched` | Selected attempt and admission boundary after validation/permission checks, before calling the handler. |
| `tool.returned`, `tool.processed` | Raw handler result and post-middleware result. |
| `tool.failed`, `tool.interrupted`, `tool.resolved` | Exception/interruption or an explicit approval/recovery decision. |
| `checkpoint`, `event`, `command.failed` | Persisted control state, extension/artifact bindings, public events, and command failures. |

The SQL backend commits supplied journal entries, state, and the corresponding
public events in the same fenced transaction. Delta-only writes append public and
private events together. Artifact bytes use existing content-addressed storage;
they are captured before references are committed, not in a cross-service atomic
transaction. Orphans are possible after a failed state commit.

Recording is enforced in the execution path, not observation hooks. A recording
commit failure blocks advancement at that boundary. When storage recovers during
error handling, pending entries retain their IDs, preventing duplicate insertion
of the same boundary. If storage remains unavailable, the run can raise
`RecordingError`; the last durable boundary remains the recovery authority.

A dispatch entry is an intent recorded before invocation. A crash immediately
after that entry cannot prove whether the handler ran. A crash after an external
write but before `tool.returned` also leaves the outcome unresolved. Existing replay
policies still apply: a `manual` tool requires resolution; `safe`/`idempotent`
policies can retry under their existing caller/adapter contracts. No exactly-once
external-effect guarantee is added. A returned error `ToolResult` records what the
handler reported; it does not independently prove that an external write failed.

Raw tool results committed before failing middleware are reused on resume. An
interrupted model call may be invoked again, with a new attempt ID. Recording a
response is not a promise of deterministic model recovery or rerun.

## Disclosure and retention

The durable journal contains raw payloads, potentially including sensitive prompts,
tool results, provider reasoning fields, and data later removed by middleware.
Enable recording only with suitable storage access and retention policies. There
is no automatic journal retention/garbage collection in this milestone.

Export defaults to **metadata only**: payloads are marked omitted and artifact
bytes are excluded. This still reveals identifiers, timing, execution boundaries,
and run status; it is not anonymization. Payload and artifact disclosure require
separate explicit flags. Common credential-shaped dictionary keys (also within
JSON strings) are redacted automatically, with redaction markers retained.

Application-specific redaction runs on a copy:

```python
def redact(kind, payload):
    if kind == "run.started":
        return None  # Omit this payload.
    return payload

policy = ExportPolicy(include_payloads=True, redact=redact)
```

Credential-key redaction is not general secret detection. Free text and binary
artifact contents need application review/policy. Binary artifact inclusion is
all-or-none for references remaining in the exported payloads; there is no binary
redactor. Removing a reference in one record does not remove it from other records.
Raw inline results may contain the same data as an omitted spill file.

Export discovers artifact references **after** redaction and records each as
included, omitted, missing, or corrupt. Spill extension snapshots map their result
IDs to artifact URIs. Only registered/captured artifacts are portable; arbitrary
filesystem paths in application strings are not automatically captured. Files
are published atomically with private permissions and without replacing an
existing destination.

## Verification and playback

The version-1 archive contains `manifest.json`, `records.jsonl`, and optional
`artifacts/<sha256>` byte members. The manifest includes SDK/Python versions, the
registered `AgentRef`, final state at export, unresolved operations, sizes, and
checksums. An inline agent's `inline:v1` label is not a code fingerprint. This
milestone does not capture dependency lockfiles or certify adapter code versions.

`.hx` is a project-specific suffix for a standard ZIP archive. Ordinary ZIP tools
can inspect it, and the reader also accepts the same bundle with a `.zip` suffix.
The internal manifest identifies the format as `harness-x.incident`, version `1`.

`verify()` returns `VerificationReport`; malformed bundles return `valid=False`.
`playback()` returns the verified manifest, ordered records, report, and included
artifact bytes, or raises `IncidentError` for a malformed bundle. It never extracts
archive members or imports bundled application code.

* **Valid** means the supported archive structure, checksums, sequence, identities,
  event serialization, artifact catalog, and recorded final-state consistency pass.
* **Complete** additionally requires the initial run record, unredacted payloads,
  all referenced artifacts, a terminal run state, and no outstanding model/tool
  attempts. This describes captured runtime boundaries, not business correctness
  or all activity inside a handler. A fully recorded failed run can be complete.
* A metadata-only, redacted, live, interrupted, or unresolved recording can remain
  useful and valid while being incomplete; `issues` explains the missing coverage.

Checksums detect corruption, not authenticity. Someone able to rewrite both data
and the manifest can recompute them. Signatures and external trust anchoring are
outside this milestone. No arbitrary semantic verification is performed.

`BundleLimits` defaults to 64 MiB for archive/expanded bytes, 32 MiB per member,
100,000 records, and 10,000 members. Oversized exports fail; they are not silently
truncated. Imports reject unknown member paths, symlinks, duplicate names/JSON keys,
encrypted members, unsupported versions, mixed run IDs, and sequence gaps. Limits
are intended for bounded incidents; export currently materializes its snapshot and
archive in memory and is not a large-history streaming exporter.

## Qualification and scope

| Backend/path | Status |
|---|---|
| SQLite local driver | Implemented and tested with process death, resume, concurrent tools, redaction, artifacts, and offline playback. |
| PostgreSQL local driver | Uses the same journal/store implementation; a live integration test is included. Requires service-backed qualification before a release claim. |
| Temporal | Recording explicitly rejected; its activity/workflow journal path is not implemented. Existing unrecorded execution remains available. |
| Offline reader | Tested in a fresh process with agent construction and socket connections forbidden. |

SQL schema migration 2 adds the journal without changing migration 1's checksum.
Existing sessions remain unrecorded. `auto_migrate=False` rejects a v1 store until
an administrator upgrades it. Downgrading an upgraded database to the old SDK
requires a migration strategy; the old schema validator will reject version 2.

Run the synthetic example with no API key or Datagol connector:

```sh
uv run python -m examples.flight_recorder --output /tmp/invoice-incident.hx
```

It injects a model connection failure, computes a synthetic overdue invoice total,
applies output middleware, destroys the temporary runtime database, then verifies
and plays back the bundle with no additional model/tool calls. Tests add a real
process death after an effect, manual reconciliation, interrupted streaming,
ambiguous storage commits, invalid archives, and artifact tampering.

This implements release-plan milestones **A–B**. Isolated step reruns, decision
validity, evidence-bound approvals, compensation, and regression generation remain
future milestones. Public symbolic memory remains deferred.