The recorder preserves model/tool attempts and their raw/processed boundaries in the durable runtime. Export a run as a portable .hx archive, verify it, and inspect the ordered timeline without executing models, handlers, bundled code, or services. It does not introduce symbolic memory or change tool signatures.

SDK usage

from harnessx import AgentRuntime, SQLiteBackend, ExportPolicy, IncidentRecorder

async with SQLiteBackend("runtime.db") as backend:
    async with AgentRuntime(agent, backend=backend, recording=True) as runtime:
        result = await runtime.run("Analyze overdue invoices")
        bundle = await runtime.export_incident(
            result.run_id,
            destination="incident.hx",  # Existing files are never overwritten.
            policy=ExportPolicy(include_payloads=True, include_artifacts=True),
        )

recorder = IncidentRecorder()
report = await recorder.verify(bundle)
playback = await recorder.playback(bundle)
for record in playback.records:
    print(record["seq"], record["kind"], record["attempt_id"])

Recording is off for new sessions unless explicitly enabled. The setting is saved with the session. On resume(), an omitted setting inherits that saved choice; an explicit incompatible setting is rejected. Recording cannot be enabled retroactively for an unrecorded run. Ordinary Agent.run() has no recorder option.

To export after a crash without resuming execution, open the store directly:

from harnessx import SQLiteBackend, export_incident, ExportPolicy

async with SQLiteBackend("runtime.db") as store:  # initialises on entry, closes on exit
    bundle = await export_incident(
        store, session_id, run_id, "crashed-run.hx",
        policy=ExportPolicy(include_payloads=True),
    )

The application must authorize access to the session before exposing either export operation. A run is checked against its supplied/bound session, but knowing those IDs is not authentication. Opening an older store with automatic migration enabled can upgrade its schema; export never starts agent execution.

Recorded boundaries and recovery

Each private journal entry has a UUID, contiguous per-run sequence, format version, session/run/step/attempt IDs, timestamp, optional execution key, and JSON payload. It is separate from the public RunEvent stream: pre-middleware responses must not leak into that stream. The journal also retains copies of public events, including stream deltas that the engine actually emits.

Boundary Meaning
run.started, run.resumed Initial request/configuration and explicit resume history.
model.started Provider-neutral request committed before provider invocation.
model.response, model.processed Normalized response before and after middleware. Provider SDK transport objects are excluded.
model.completed, model.failed, model.interrupted Engine transition, captured failure, or interruption. Retries have distinct attempt IDs.
tool.started, tool.dispatched Selected attempt and admission boundary after validation/permission checks, before calling the handler.
tool.returned, tool.processed Raw handler result and post-middleware result.
tool.failed, tool.interrupted, tool.resolved Exception/interruption or an explicit approval/recovery decision.
checkpoint, event, command.failed Persisted control state, extension/artifact bindings, public events, and command failures.

The SQL backend commits supplied journal entries, state, and the corresponding public events in the same fenced transaction. Delta-only writes append public and private events together. Artifact bytes use existing content-addressed storage; they are captured before references are committed, not in a cross-service atomic transaction. Orphans are possible after a failed state commit.

Recording is enforced in the execution path, not observation hooks. A recording commit failure blocks advancement at that boundary. When storage recovers during error handling, pending entries retain their IDs, preventing duplicate insertion of the same boundary. If storage remains unavailable, the run can raise RecordingError; the last durable boundary remains the recovery authority.

A dispatch entry is an intent recorded before invocation. A crash immediately after that entry cannot prove whether the handler ran. A crash after an external write but before tool.returned also leaves the outcome unresolved. Existing replay policies still apply: a manual tool requires resolution; safe/idempotent policies can retry under their existing caller/adapter contracts. No exactly-once external-effect guarantee is added. A returned error ToolResult records what the handler reported; it does not independently prove that an external write failed.

Raw tool results committed before failing middleware are reused on resume. An interrupted model call may be invoked again, with a new attempt ID. Recording a response is not a promise of deterministic model recovery or rerun.

Disclosure and retention

The durable journal contains raw payloads, potentially including sensitive prompts, tool results, provider reasoning fields, and data later removed by middleware. Enable recording only with suitable storage access and retention policies. There is no automatic journal retention/garbage collection in this milestone.

Export defaults to metadata only: payloads are marked omitted and artifact bytes are excluded. This still reveals identifiers, timing, execution boundaries, and run status; it is not anonymization. Payload and artifact disclosure require separate explicit flags. Common credential-shaped dictionary keys (also within JSON strings) are redacted automatically, with redaction markers retained.

Application-specific redaction runs on a copy:

def redact(kind, payload):
    if kind == "run.started":
        return None  # Omit this payload.
    return payload

policy = ExportPolicy(include_payloads=True, redact=redact)

Credential-key redaction is not general secret detection. Free text and binary artifact contents need application review/policy. Binary artifact inclusion is all-or-none for references remaining in the exported payloads; there is no binary redactor. Removing a reference in one record does not remove it from other records. Raw inline results may contain the same data as an omitted spill file.

Export discovers artifact references after redaction and records each as included, omitted, missing, or corrupt. Spill extension snapshots map their result IDs to artifact URIs. Only registered/captured artifacts are portable; arbitrary filesystem paths in application strings are not automatically captured. Files are published atomically with private permissions and without replacing an existing destination.

Verification and playback

The version-1 archive contains manifest.json, records.jsonl, and optional artifacts/<sha256> byte members. The manifest includes SDK/Python versions, the registered AgentRef, final state at export, unresolved operations, sizes, and checksums. An inline agent's inline:v1 label is not a code fingerprint. This milestone does not capture dependency lockfiles or certify adapter code versions.

.hx is a project-specific suffix for a standard ZIP archive. Ordinary ZIP tools can inspect it, and the reader also accepts the same bundle with a .zip suffix. The internal manifest identifies the format as harness-x.incident, version 1.

verify() returns VerificationReport; malformed bundles return valid=False. playback() returns the verified manifest, ordered records, report, and included artifact bytes, or raises IncidentError for a malformed bundle. It never extracts archive members or imports bundled application code.

  • Valid means the supported archive structure, checksums, sequence, identities, event serialization, artifact catalog, and recorded final-state consistency pass.
  • Complete additionally requires the initial run record, unredacted payloads, all referenced artifacts, a terminal run state, and no outstanding model/tool attempts. This describes captured runtime boundaries, not business correctness or all activity inside a handler. A fully recorded failed run can be complete.
  • A metadata-only, redacted, live, interrupted, or unresolved recording can remain useful and valid while being incomplete; issues explains the missing coverage.

Checksums detect corruption, not authenticity. Someone able to rewrite both data and the manifest can recompute them. Signatures and external trust anchoring are outside this milestone. No arbitrary semantic verification is performed.

BundleLimits defaults to 64 MiB for archive/expanded bytes, 32 MiB per member, 100,000 records, and 10,000 members. Oversized exports fail; they are not silently truncated. Imports reject unknown member paths, symlinks, duplicate names/JSON keys, encrypted members, unsupported versions, mixed run IDs, and sequence gaps. Limits are intended for bounded incidents; export currently materializes its snapshot and archive in memory and is not a large-history streaming exporter.

Qualification and scope

Backend/path Status
SQLite local driver Implemented and tested with process death, resume, concurrent tools, redaction, artifacts, and offline playback.
PostgreSQL local driver Uses the same journal/store implementation; a live integration test is included. Requires service-backed qualification before a release claim.
Temporal Recording explicitly rejected; its activity/workflow journal path is not implemented. Existing unrecorded execution remains available.
Offline reader Tested in a fresh process with agent construction and socket connections forbidden.

SQL schema migration 2 adds the journal without changing migration 1's checksum. Existing sessions remain unrecorded. auto_migrate=False rejects a v1 store until an administrator upgrades it. Downgrading an upgraded database to the old SDK requires a migration strategy; the old schema validator will reject version 2.

Run the synthetic example with no API key or Datagol connector:

uv run python -m examples.flight_recorder --output /tmp/invoice-incident.hx

It injects a model connection failure, computes a synthetic overdue invoice total, applies output middleware, destroys the temporary runtime database, then verifies and plays back the bundle with no additional model/tool calls. Tests add a real process death after an effect, manual reconciliation, interrupted streaming, ambiguous storage commits, invalid archives, and artifact tampering.

This implements release-plan milestones A–B. Isolated step reruns, decision validity, evidence-bound approvals, compensation, and regression generation remain future milestones. Public symbolic memory remains deferred.