# PostgreSQL walkthrough Write a report, crash, and resume the same run without repeating a committed tool call. This walkthrough is part of **Advanced topics**. Read [Durable agent runs](../durable-agent-runs/) for session IDs, RunHandle, RunResult, progress events, approvals, and the Agent/AgentRef distinction. Imagine an agent preparing a weekly sales report. It writes the report, then the Python process crashes before it can tell you the result. Starting a new ordinary agent would not automatically recover that unfinished run. With `AgentRuntime` and PostgreSQL, the new process can load the saved run and continue from its last committed execution boundary. In this example, the report is written **once**. After the crash, another Python process finishes the same run using the tool result already stored in PostgreSQL. Process 1asks, writes, crashes PostgreSQLruns · tool results Local filesthe report on disk Ask for the report saves the run and the tool intent Run write_report report.txt written Save the tool result committed before anything else happens CRASH · EXIT CODE 42 Process 2a new Python process resume(the same session) the run and its saved result come back Reuse the result, finish the answer marks the same run completed report.txt unchanged WHAT SURVIVED THE CRASH The sessionsame session_id, resumed by name The runone run, completed once The tool callwrite_report executed exactly once
  1. Process 1 asks for the report; PostgreSQL saves the run and the tool intent.
  2. write_report runs and writes report.txt.
  3. PostgreSQL saves the tool result, committed before anything else happens.
  4. Process 1 crashes with exit code 42.
  5. Process 2 resumes the same session; the run and its saved result come back.
  6. The result is reused, the answer is finished, and PostgreSQL marks the same run completed. report.txt is unchanged.
The model responses are scripted so the demonstration always reaches the chosen crash boundary. It uses the real execution engine, tool dispatcher, and PostgreSQL backend. **No model API key or paid model call is needed** for `crash`, `status`, `resume`, or `check`. The separate `chat` command uses a live model. ## 1. Connect to PostgreSQL From a clone of the repository, install the PostgreSQL extra into the environment uv manages: ```bash uv sync --extra postgres ``` Prefix every command below with `uv run` so they share that environment. The examples live in the checkout, not in the installed package. Use `examples/postgres.config.json` for your local connection settings. It is Git-ignored. If that file has not been prepared in your checkout, copy [`examples/postgres.config.example.json`](https://github.com/datagol/agent-harness-x/blob/main/examples/postgres.config.example.json) to that name and fill in your server, database, and username: ```json { "connection_string": "postgresql://YOUR_USER:{password}@YOUR_SERVER.postgres.database.azure.com:5432/YOUR_DATABASE?sslmode=require&connect_timeout=10", "password": "REPLACE_WITH_PASSWORD" } ``` Replace `REPLACE_WITH_PASSWORD` in the **password field** with your actual password. Leave `{password}` in the connection string. The example encodes the password when inserting it into the URL, so characters such as `@`, `/`, and `%` work without manual URL encoding. Keep valid JSON syntax; escape embedded double quotes or backslashes as `\"` and `\\`. The template requests TLS with `sslmode=require`, following the [Azure PostgreSQL Python connection guide](https://learn.microsoft.com/en-us/azure/postgresql/connectivity/connect-python). It also sets a 10-second connection timeout. After filling in the password, check the connection: ```bash .venv/bin/uv run python -m examples.postgres_runtime check --config examples/postgres.config.json ``` `check` connects and provisions or validates the HarnessX runtime schema. It does not invoke a model or create an agent session. Success prints: ```text Connected to PostgreSQL. HarnessX schema 'harness_x' is ready. ``` To run an actual live-model agent against this database, with `ANTHROPIC_API_KEY` set in your environment or the repository's `.env`, run: ```bash .venv/bin/uv run python -m examples.postgres_runtime chat --config examples/postgres.config.json ``` For the controlled crash/recovery demonstration, continue with the commands below. Alternatively, set a connection string in your shell or the repository's `.env`: ```bash export DATABASE_URL='postgresql://USER:PASSWORD@HOST:5432/DATABASE' ``` When using `DATABASE_URL`, omit `--config` from the commands. An explicit `--config` takes precedence over `DATABASE_URL`; otherwise shell variables take precedence over `.env`. The example rejects an unfilled config password before attempting a connection and does not print the connection string or password. Replace URL placeholders with your actual connection details and preserve the TLS options required by your database provider. The backend creates HarnessX's schema and tables; it does not create a PostgreSQL server, database, or user. The database role needs permission to create the schema and its objects on first use. The default schema is `harness_x`, retained for existing database compatibility even though the Python package is `harnessx`. If you already have a local PostgreSQL installation, use a database on that installation. If you do not have a server yet, create one before continuing. ## 2. Write the report and crash ```bash .venv/bin/uv run python -m examples.postgres_runtime crash --config examples/postgres.config.json ``` Expected output, with different session IDs and paths: ```text Session: Workspace: /.agent_sessions/postgres-demo write_report executed: report.txt created Tool result committed. Intentionally exiting this process (code 42). Wait 30 seconds for the lease to expire, then run the resume command. ``` Exit code **42 is intentional**. This is an abrupt process exit that bypasses Python cleanup; it is not a normal pause or an exception caught by the agent. The process is gone. There is no hidden agent continuing in the background. The example installs fault-injection middleware to exit immediately after the engine saves the raw tool result and before result processing completes. Do not install that middleware in an application agent. Durability itself is enforced by the execution engine and backend, not by this middleware. Inspect the local report and invocation log: ```bash cat .agent_sessions/postgres-demo/report.txt cat .agent_sessions/postgres-demo/tool-invocations.txt ``` The report contains 12 orders and $1,200 revenue. The log contains **one** `write_report` line. Every invocation appends a line before attempting the write, so a repeated invocation would be visible even if exclusive file creation failed. ## 3. See what PostgreSQL remembers ```bash .venv/bin/uv run python -m examples.postgres_runtime status --config examples/postgres.config.json ``` Expected output: ```text Session: Run: Status: running Tool: write_report | Saved state: raw_completed Report exists: True Tool invocations: 1 ``` `running` is the **last persisted run status**, not proof that a process is alive. The crashed worker could not write a final status. `raw_completed` means the tool returned and its result was committed, but result processing is unfinished. `status` only reads saved state and local files; it does not claim the session or resume work. ## 4. Restart and finish Wait at least **30 seconds after the crash**, then run: ```bash .venv/bin/uv run python -m examples.postgres_runtime resume --config examples/postgres.config.json ``` Expected output includes: ```text Session: Run: Status: completed Report exists: True Tool invocations: 1 Answer: The weekly report is ready: 12 orders, $1,200 revenue. ``` The report tool is not called again. The new process restores the conversation, finishes processing the saved tool result, and obtains the final scripted answer. It uses the **same session and run IDs**, not a new run with a similar prompt. Run `resume` again: it prints `No unfinished run; showing saved results.` The invocation count remains 1. The example checks both the count and report contents and fails if either is wrong. The wait exists because PostgreSQL gives an executing worker a lease. The default lease lasts 30 seconds and is renewed about every 10 seconds. A hard crash stops renewal but cannot release the lease. A replacement worker must wait for it to expire. If you resume too soon, the example prints an actionable message; retry after the wait. Do not force-clear an active worker's lease. ## Files, IDs, and repeat runs | Location | Contents | Purpose | | --- | --- | --- | | PostgreSQL | Session definition, run state, messages, model/tool progress, saved results, events | Execution recovery | | `session.json` in the workspace | Session ID and PostgreSQL schema name | Lets the next CLI command find the same session; no credentials or execution state | | `report.txt` in the workspace | The report created by the tool | The visible tool effect | | `tool-invocations.txt` in the workspace | One line per tool invocation | Detects accidental repetition | The default workspace is `.agent_sessions/postgres-demo`. For a fresh test, choose a new directory and use it in **all three** commands: ```bash .venv/bin/uv run python -m examples.postgres_runtime crash --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2 .venv/bin/uv run python -m examples.postgres_runtime status --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2 # Wait at least 30 seconds after the crash. .venv/bin/uv run python -m examples.postgres_runtime resume --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2 ``` The example refuses to start over in a workspace containing demo files. PostgreSQL records are retained for inspection. A new workspace gets a new session; changing the local folder does not delete database records. `--schema harness_demo` on the `crash` command selects a separate demo schema; subsequent commands read that selection from `session.json`. Keep your config or `DATABASE_URL` pointed at the same database; credentials are not copied into the session file. Local files do **not** become PostgreSQL artifacts automatically. This example assumes all commands run on the same machine with the same workspace. If the report is deleted, its saved tool result still exists, but the report itself is gone; the example detects the missing file. For recovery on another machine, store outputs in a shared service or an explicitly configured artifact store and bind compatible tools on the replacement worker. ## Use an Agent directly For PostgreSQL, create your agent and pass it to `AgentRuntime(agent, backend=backend)`. No `AgentRegistry` or `AgentRef` is needed. The following functions show ordinary live-model usage; unlike the scripted CLI demo, they require the default Anthropic provider's API key. Start a new session: ```python from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend async def start_session(dsn: str): async with PostgresBackend(dsn) as backend: # opens the pool and migrates on entry agent = Agent(config=AgentConfig(system_prompt="You are helpful.")) async with agent, AgentRuntime(agent, backend=backend) as runtime: session_id = runtime.session_id print("Session:", session_id) # Store this ID in your application. result = await runtime.run("Hello", request_id="welcome-1") return session_id, result ``` Call `await start_session(dsn)` from your async application. Store the session ID as soon as the session starts so a replacement process can find it after a crash. In the replacement process, construct a **fresh** agent with the same configuration and tools, then resume the saved session: ```python from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend async def resume_session(dsn: str, saved_session_id: str): async with PostgresBackend(dsn) as backend: agent = Agent(config=AgentConfig(system_prompt="You are helpful.")) async with agent: runtime = AgentRuntime(agent, backend=backend) try: handle = await runtime.resume(saved_session_id) if handle: return await handle.result() return None # No unfinished run to drive. finally: await runtime.stop() ``` Call `await resume_session(dsn, saved_session_id)` with the original database and session ID. The start function above normally completes before returning, so resuming that completed session returns `None`. Use the CLI crash demonstration to observe unfinished work being recovered. A resumed run can also return `awaiting_input` if an approval or uncertain tool outcome needs resolution. The outer agent context ensures cleanup even if connecting or resuming fails before the runtime binds the agent. A runtime closes the agent it executes; repeated close calls are safe. Since the application supplies the backend, it also closes that backend: leaving `async with PostgresBackend(dsn)` does so, and `await PostgresBackend.connect(dsn)` constructs and initialises a backend in one call when a context manager does not fit. A closed agent should not be reused for another runtime. ## How the crash demonstration uses this API The complete runnable implementation is [`examples/postgres_runtime.py`](https://github.com/datagol/agent-harness-x/blob/main/examples/postgres_runtime.py). Its main pieces are: 1. `report_agent(workspace, crash=...)` is an ordinary helper that constructs an `Agent` with the report tool and scripted provider. Each CLI process creates its own instance. Python functions and credentials are supplied by your application; they are not stored in PostgreSQL. 2. `PostgresBackend(DATABASE_URL)` stores execution state in the configured database. `AgentRuntime(agent, backend=backend)` connects that storage to the configured agent. 3. `await runtime.start()` creates a session. The demo saves the returned ID in `session.json`. A session is a conversation; each submitted request creates a run within that conversation. 4. `await runtime.run(prompt, request_id="weekly-report")` starts this run. A request ID deduplicates the same request **within the same session**. It neither finds a session after restart nor deduplicates arbitrary external effects. 5. In the replacement process, `await runtime.resume(session_id)` restores and drives unfinished work. `await handle.result()` waits for its result. If no unfinished work exists, `resume` returns `None`. 6. `await runtime.stop()` (or its alias `aclose()`) closes the runtime's agent, and `await backend.aclose()` closes the caller-owned backend. The intentional crash skips both; the recovery process performs normal cleanup. For a new session, the async context manager calls `start()` automatically. For recovery, construct the runtime and call `resume(session_id)` directly; entering `async with AgentRuntime(...)` first would create a new session. Use matching configuration and compatible tools and extensions when resuming. The runtime compares the agent configuration with the saved configuration; it does not prove that changed Python handlers have the same behavior. Your application chooses the correct agent implementation for each saved session. `AgentRegistry` and `AgentRef` remain supported for named definitions and Temporal workers. Sessions started using a named reference must be resumed with the same name/version and a compatible registered factory. Passing an `Agent` directly does not convert an existing named session. If you ran an older registry-based version of this demo, choose a fresh `--workspace` for this version. ## The crash boundary matters The report tool uses the default `manual` replay policy. That does **not** prevent reuse of a committed result. It prevents automatic repetition when the runtime cannot establish what happened. | Crash boundary | What is known | Recovery behavior | | --- | --- | --- | | Before dispatch, with intent saved | The tool has not been started by the runtime | Execute the prepared call | | After dispatch is recorded, before the raw result commits | The tool may or may not have produced an effect | A `manual` call becomes uncertain; the run waits for reconciliation | | After the raw result commits, as in this demo | The returned tool result is saved | Reuse it and retry unfinished result processing; do not invoke the tool again | | After the run completes | The final answer is saved | `resume` returns `None`; inspect the saved result | The difficult case is a tool that sends an email, charges a card, or creates a record, followed by a crash **before HarnessX saves the result**. The external system and PostgreSQL are not one transaction. HarnessX cannot infer whether the effect happened. Investigate the external system before choosing a resolution: - Supply a verified `ToolResult` through `runtime.resolve_tool(...)`. - Explicitly request a retry, accepting the possibility of repetition. - Abort the run. See [approvals and recovery](../recovery/) for the API. A replacement runtime must first bind/resume the saved session so these operations target that session. Resolve the pending execution key and resume again to continue. Ordinary tool exceptions can be returned as error results; do not assume every remote exception is automatically classified as uncertain. For read-only operations that can safely repeat, declare `replay_policy="safe"`. For an effect whose external service enforces an idempotency key, declare `replay_policy="idempotent"` and pass `current_tool_context().execution_key` to that service. The policy name alone does not implement idempotency. Default to `manual` for uncertain effects. ## What PostgreSQL does and does not provide Each SQL execution-boundary save commits the run state and its associated events in a transaction. Stored conversation snapshots, approvals, tool outcomes, and artifact data support recovery; optional flight recording adds a separate incident journal. Stored artifacts and external effects do not share one global transaction. Leases prevent two cooperative workers from driving the same session at once. Fencing rejects state writes from a worker that has lost its lease. It cannot stop an already-running external request from completing. Different sessions can run concurrently; external systems still need their own concurrency controls. PostgreSQL is storage, **not a worker scheduler**. Your application, job runner, or supervisor must restart a worker and call `resume` for interrupted sessions. Database availability, backups, storage durability settings, and failover policy remain deployment responsibilities. This example demonstrates a worker-process crash, not a database crash, regional outage, or failover test. An interrupted live model call may need to be issued again, incur another charge, and produce different output. The scripted provider makes this particular demo predictable; live models do not become deterministic. Result middleware may also run again, so keep its side effects replay-safe. ## Inspect the database directly For the default schema, run these read-only queries in your SQL client, replacing the IDs with those printed by `status`: ```sql SELECT id, session_id, status FROM harness_x.runs WHERE session_id = 'COPY_SESSION_ID'; SELECT payload->>'execution_key' AS execution_key, payload->>'status' AS tool_status, payload->'raw_result'->>'content' AS saved_result FROM harness_x.tool_outcomes WHERE run_id = 'COPY_RUN_ID'; ``` Before recovery, the saved tool status is `raw_completed`; after recovery it is `completed`. Runtime tables and JSON payloads are storage internals, not a stable application API. Use `AgentRuntime` to change execution state. Do not update table rows to approve tools, release leases, or mark work complete. ## Troubleshooting and verification | Symptom | What to check | | --- | --- | | `Install harnessx[postgres]` | Install the extra into the same interpreter used to run the example | | `Set DATABASE_URL or pass --config` | Pass your JSON config path, export `DATABASE_URL`, or set it in the checkout's `.env` | | `Set the password field` | Replace `REPLACE_WITH_PASSWORD` in your local config; leave `{password}` in its connection string | | Config cannot be read | Check the `--config` path and JSON syntax; secrets are omitted from parse errors | | Initialization failure | Server reachability, database name, credentials, TLS, and schema privileges | | Exit code 42 from `crash` | Expected fault injection; continue with `status`, then `resume` | | Active lease on `resume` | Wait at least 30 seconds after the crash; another worker may also own the session | | `No demo session found` | Use the same workspace and working directory as the `crash` command | | Session not found in storage | Check that your config or `DATABASE_URL` still points at the original database | | Demo files already exist | Resume them or select a fresh `--workspace`; rerunning `crash` does not start over | | `awaiting_input` after a different crash | Inspect pending approvals or uncertain effects; do not blindly replay | The automated demo test launches separate Python processes, checks the exit code and saved raw result, resumes, and verifies that the run ID and single tool invocation are preserved. The default test uses SQLite as an explicit stand-in for the SQL storage path: ```bash .venv/bin/python -m pytest tests/test_postgres_example.py -q ``` To run that same process-crash scenario against PostgreSQL, plus the existing backend tests, configure a **test database** whose role can create/drop isolated test schemas: ```bash HARNESS_TEST_POSTGRES_DSN="$DATABASE_URL" .venv/bin/python -m pytest \ tests/test_postgres_example.py tests/test_postgres_runtime.py \ -k 'not missing_schema_privilege' -v ``` The excluded privilege test additionally needs permission to create roles. Tests use unique disposable schemas and clean up those schemas. The PostgreSQL crash test shortens its lease to two seconds; the CLI uses the normal 30-second default. Local subprocess recovery and the live PostgreSQL integration suite are separate claims. Passing with a SQLite stand-in does not qualify PostgreSQL pooling, locking, schema provisioning, or your deployment's failover behavior. PostgreSQL remains subject to service-backed release qualification.