PostgreSQL walkthrough

This walkthrough is part of Advanced topics. Read Durable agent runs for session IDs, RunHandle, RunResult, progress events, approvals, and the Agent/AgentRef distinction.

Imagine an agent preparing a weekly sales report. It writes the report, then the Python process crashes before it can tell you the result. Starting a new ordinary agent would not automatically recover that unfinished run. With AgentRuntime and PostgreSQL, the new process can load the saved run and continue from its last committed execution boundary.

In this example, the report is written once. After the crash, another Python process finishes the same run using the tool result already stored in PostgreSQL.

The model responses are scripted so the demonstration always reaches the chosen crash boundary. It uses the real execution engine, tool dispatcher, and PostgreSQL backend. No model API key or paid model call is needed for crash, status, resume, or check. The separate chat command uses a live model.

1. Connect to PostgreSQL

From a clone of the repository, install the PostgreSQL extra into the environment uv manages:

uv sync --extra postgres

Prefix every command below with uv run so they share that environment. The examples live in the checkout, not in the installed package.

Use examples/postgres.config.json for your local connection settings. It is Git-ignored. If that file has not been prepared in your checkout, copy examples/postgres.config.example.json to that name and fill in your server, database, and username:

{
  "connection_string": "postgresql://YOUR_USER:{password}@YOUR_SERVER.postgres.database.azure.com:5432/YOUR_DATABASE?sslmode=require&connect_timeout=10",
  "password": "REPLACE_WITH_PASSWORD"
}

Replace REPLACE_WITH_PASSWORD in the password field with your actual password. Leave {password} in the connection string. The example encodes the password when inserting it into the URL, so characters such as @, /, and % work without manual URL encoding. Keep valid JSON syntax; escape embedded double quotes or backslashes as \" and \\.

The template requests TLS with sslmode=require, following the Azure PostgreSQL Python connection guide. It also sets a 10-second connection timeout.

After filling in the password, check the connection:

.venv/bin/uv run python -m examples.postgres_runtime check --config examples/postgres.config.json

check connects and provisions or validates the HarnessX runtime schema. It does not invoke a model or create an agent session. Success prints:

Connected to PostgreSQL. HarnessX schema 'harness_x' is ready.

To run an actual live-model agent against this database, with ANTHROPIC_API_KEY set in your environment or the repository's .env, run:

.venv/bin/uv run python -m examples.postgres_runtime chat --config examples/postgres.config.json

For the controlled crash/recovery demonstration, continue with the commands below.

Alternatively, set a connection string in your shell or the repository's .env:

export DATABASE_URL='postgresql://USER:PASSWORD@HOST:5432/DATABASE'

When using DATABASE_URL, omit --config from the commands. An explicit --config takes precedence over DATABASE_URL; otherwise shell variables take precedence over .env. The example rejects an unfilled config password before attempting a connection and does not print the connection string or password.

Replace URL placeholders with your actual connection details and preserve the TLS options required by your database provider. The backend creates HarnessX's schema and tables; it does not create a PostgreSQL server, database, or user. The database role needs permission to create the schema and its objects on first use. The default schema is harness_x, retained for existing database compatibility even though the Python package is harnessx.

If you already have a local PostgreSQL installation, use a database on that installation. If you do not have a server yet, create one before continuing.

2. Write the report and crash

.venv/bin/uv run python -m examples.postgres_runtime crash --config examples/postgres.config.json

Expected output, with different session IDs and paths:

Session: <session-id>
Workspace: <repository>/.agent_sessions/postgres-demo
write_report executed: report.txt created
Tool result committed. Intentionally exiting this process (code 42).
Wait 30 seconds for the lease to expire, then run the resume command.

Exit code 42 is intentional. This is an abrupt process exit that bypasses Python cleanup; it is not a normal pause or an exception caught by the agent. The process is gone. There is no hidden agent continuing in the background.

The example installs fault-injection middleware to exit immediately after the engine saves the raw tool result and before result processing completes. Do not install that middleware in an application agent. Durability itself is enforced by the execution engine and backend, not by this middleware.

Inspect the local report and invocation log:

cat .agent_sessions/postgres-demo/report.txt
cat .agent_sessions/postgres-demo/tool-invocations.txt

The report contains 12 orders and $1,200 revenue. The log contains one write_report line. Every invocation appends a line before attempting the write, so a repeated invocation would be visible even if exclusive file creation failed.

3. See what PostgreSQL remembers

.venv/bin/uv run python -m examples.postgres_runtime status --config examples/postgres.config.json

Expected output:

Session: <same-session-id>
Run: <run-id>
Status: running
Tool: write_report | Saved state: raw_completed
Report exists: True
Tool invocations: 1

running is the last persisted run status, not proof that a process is alive. The crashed worker could not write a final status. raw_completed means the tool returned and its result was committed, but result processing is unfinished. status only reads saved state and local files; it does not claim the session or resume work.

4. Restart and finish

Wait at least 30 seconds after the crash, then run:

.venv/bin/uv run python -m examples.postgres_runtime resume --config examples/postgres.config.json

Expected output includes:

Session: <same-session-id>
Run: <same-run-id>
Status: completed
Report exists: True
Tool invocations: 1
Answer: The weekly report is ready: 12 orders, $1,200 revenue.

The report tool is not called again. The new process restores the conversation, finishes processing the saved tool result, and obtains the final scripted answer. It uses the same session and run IDs, not a new run with a similar prompt.

Run resume again: it prints No unfinished run; showing saved results. The invocation count remains 1. The example checks both the count and report contents and fails if either is wrong.

The wait exists because PostgreSQL gives an executing worker a lease. The default lease lasts 30 seconds and is renewed about every 10 seconds. A hard crash stops renewal but cannot release the lease. A replacement worker must wait for it to expire. If you resume too soon, the example prints an actionable message; retry after the wait. Do not force-clear an active worker's lease.

Files, IDs, and repeat runs

Location	Contents	Purpose
PostgreSQL	Session definition, run state, messages, model/tool progress, saved results, events	Execution recovery
session.json in the workspace	Session ID and PostgreSQL schema name	Lets the next CLI command find the same session; no credentials or execution state
report.txt in the workspace	The report created by the tool	The visible tool effect
tool-invocations.txt in the workspace	One line per tool invocation	Detects accidental repetition

The default workspace is .agent_sessions/postgres-demo. For a fresh test, choose a new directory and use it in all three commands:

.venv/bin/uv run python -m examples.postgres_runtime crash --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2
.venv/bin/uv run python -m examples.postgres_runtime status --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2
# Wait at least 30 seconds after the crash.
.venv/bin/uv run python -m examples.postgres_runtime resume --config examples/postgres.config.json --workspace .agent_sessions/pg-demo-2

The example refuses to start over in a workspace containing demo files. PostgreSQL records are retained for inspection. A new workspace gets a new session; changing the local folder does not delete database records. --schema harness_demo on the crash command selects a separate demo schema; subsequent commands read that selection from session.json. Keep your config or DATABASE_URL pointed at the same database; credentials are not copied into the session file.

Local files do not become PostgreSQL artifacts automatically. This example assumes all commands run on the same machine with the same workspace. If the report is deleted, its saved tool result still exists, but the report itself is gone; the example detects the missing file. For recovery on another machine, store outputs in a shared service or an explicitly configured artifact store and bind compatible tools on the replacement worker.

Use an Agent directly

For PostgreSQL, create your agent and pass it to AgentRuntime(agent, backend=backend). No AgentRegistry or AgentRef is needed. The following functions show ordinary live-model usage; unlike the scripted CLI demo, they require the default Anthropic provider's API key.

Start a new session:

from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend


async def start_session(dsn: str):
    async with PostgresBackend(dsn) as backend:  # opens the pool and migrates on entry
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent, AgentRuntime(agent, backend=backend) as runtime:
            session_id = runtime.session_id
            print("Session:", session_id)  # Store this ID in your application.
            result = await runtime.run("Hello", request_id="welcome-1")
            return session_id, result

Call await start_session(dsn) from your async application. Store the session ID as soon as the session starts so a replacement process can find it after a crash.

In the replacement process, construct a fresh agent with the same configuration and tools, then resume the saved session:

from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend


async def resume_session(dsn: str, saved_session_id: str):
    async with PostgresBackend(dsn) as backend:
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent:
            runtime = AgentRuntime(agent, backend=backend)
            try:
                handle = await runtime.resume(saved_session_id)
                if handle:
                    return await handle.result()
                return None  # No unfinished run to drive.
            finally:
                await runtime.stop()

Call await resume_session(dsn, saved_session_id) with the original database and session ID. The start function above normally completes before returning, so resuming that completed session returns None. Use the CLI crash demonstration to observe unfinished work being recovered. A resumed run can also return awaiting_input if an approval or uncertain tool outcome needs resolution.

The outer agent context ensures cleanup even if connecting or resuming fails before the runtime binds the agent. A runtime closes the agent it executes; repeated close calls are safe. Since the application supplies the backend, it also closes that backend: leaving async with PostgresBackend(dsn) does so, and await PostgresBackend.connect(dsn) constructs and initialises a backend in one call when a context manager does not fit. A closed agent should not be reused for another runtime.

How the crash demonstration uses this API

The complete runnable implementation is examples/postgres_runtime.py. Its main pieces are:

1. report_agent(workspace, crash=...) is an ordinary helper that constructs an Agent with the report tool and scripted provider. Each CLI process creates its own instance. Python functions and credentials are supplied by your application; they are not stored in PostgreSQL.
2. PostgresBackend(DATABASE_URL) stores execution state in the configured database. AgentRuntime(agent, backend=backend) connects that storage to the configured agent.
3. await runtime.start() creates a session. The demo saves the returned ID in session.json. A session is a conversation; each submitted request creates a run within that conversation.
4. await runtime.run(prompt, request_id="weekly-report") starts this run. A request ID deduplicates the same request within the same session. It neither finds a session after restart nor deduplicates arbitrary external effects.
5. In the replacement process, await runtime.resume(session_id) restores and drives unfinished work. await handle.result() waits for its result. If no unfinished work exists, resume returns None.
6. await runtime.stop() (or its alias aclose()) closes the runtime's agent, and await backend.aclose() closes the caller-owned backend. The intentional crash skips both; the recovery process performs normal cleanup.

For a new session, the async context manager calls start() automatically. For recovery, construct the runtime and call resume(session_id) directly; entering async with AgentRuntime(...) first would create a new session.

Use matching configuration and compatible tools and extensions when resuming. The runtime compares the agent configuration with the saved configuration; it does not prove that changed Python handlers have the same behavior. Your application chooses the correct agent implementation for each saved session.

AgentRegistry and AgentRef remain supported for named definitions and Temporal workers. Sessions started using a named reference must be resumed with the same name/version and a compatible registered factory. Passing an Agent directly does not convert an existing named session. If you ran an older registry-based version of this demo, choose a fresh --workspace for this version.

The crash boundary matters

The report tool uses the default manual replay policy. That does not prevent reuse of a committed result. It prevents automatic repetition when the runtime cannot establish what happened.

Crash boundary	What is known	Recovery behavior
Before dispatch, with intent saved	The tool has not been started by the runtime	Execute the prepared call
After dispatch is recorded, before the raw result commits	The tool may or may not have produced an effect	A manual call becomes uncertain; the run waits for reconciliation
After the raw result commits, as in this demo	The returned tool result is saved	Reuse it and retry unfinished result processing; do not invoke the tool again
After the run completes	The final answer is saved	resume returns None; inspect the saved result

The difficult case is a tool that sends an email, charges a card, or creates a record, followed by a crash before HarnessX saves the result. The external system and PostgreSQL are not one transaction. HarnessX cannot infer whether the effect happened. Investigate the external system before choosing a resolution:

• Supply a verified ToolResult through runtime.resolve_tool(...).
• Explicitly request a retry, accepting the possibility of repetition.
• Abort the run.

See approvals and recovery for the API. A replacement runtime must first bind/resume the saved session so these operations target that session. Resolve the pending execution key and resume again to continue. Ordinary tool exceptions can be returned as error results; do not assume every remote exception is automatically classified as uncertain.

For read-only operations that can safely repeat, declare replay_policy="safe". For an effect whose external service enforces an idempotency key, declare replay_policy="idempotent" and pass current_tool_context().execution_key to that service. The policy name alone does not implement idempotency. Default to manual for uncertain effects.

What PostgreSQL does and does not provide

Each SQL execution-boundary save commits the run state and its associated events in a transaction. Stored conversation snapshots, approvals, tool outcomes, and artifact data support recovery; optional flight recording adds a separate incident journal. Stored artifacts and external effects do not share one global transaction.

Leases prevent two cooperative workers from driving the same session at once. Fencing rejects state writes from a worker that has lost its lease. It cannot stop an already-running external request from completing. Different sessions can run concurrently; external systems still need their own concurrency controls.

PostgreSQL is storage, not a worker scheduler. Your application, job runner, or supervisor must restart a worker and call resume for interrupted sessions. Database availability, backups, storage durability settings, and failover policy remain deployment responsibilities. This example demonstrates a worker-process crash, not a database crash, regional outage, or failover test.

An interrupted live model call may need to be issued again, incur another charge, and produce different output. The scripted provider makes this particular demo predictable; live models do not become deterministic. Result middleware may also run again, so keep its side effects replay-safe.

Inspect the database directly

For the default schema, run these read-only queries in your SQL client, replacing the IDs with those printed by status:

SELECT id, session_id, status
FROM harness_x.runs
WHERE session_id = 'COPY_SESSION_ID';

SELECT payload->>'execution_key' AS execution_key,
       payload->>'status' AS tool_status,
       payload->'raw_result'->>'content' AS saved_result
FROM harness_x.tool_outcomes
WHERE run_id = 'COPY_RUN_ID';

Before recovery, the saved tool status is raw_completed; after recovery it is completed. Runtime tables and JSON payloads are storage internals, not a stable application API. Use AgentRuntime to change execution state. Do not update table rows to approve tools, release leases, or mark work complete.

Troubleshooting and verification

Symptom	What to check
Install harnessx[postgres]	Install the extra into the same interpreter used to run the example
Set DATABASE_URL or pass --config	Pass your JSON config path, export DATABASE_URL, or set it in the checkout's .env
Set the password field	Replace REPLACE_WITH_PASSWORD in your local config; leave {password} in its connection string
Config cannot be read	Check the --config path and JSON syntax; secrets are omitted from parse errors
Initialization failure	Server reachability, database name, credentials, TLS, and schema privileges
Exit code 42 from crash	Expected fault injection; continue with status, then resume
Active lease on resume	Wait at least 30 seconds after the crash; another worker may also own the session
No demo session found	Use the same workspace and working directory as the crash command
Session not found in storage	Check that your config or DATABASE_URL still points at the original database
Demo files already exist	Resume them or select a fresh --workspace; rerunning crash does not start over
awaiting_input after a different crash	Inspect pending approvals or uncertain effects; do not blindly replay

The automated demo test launches separate Python processes, checks the exit code and saved raw result, resumes, and verifies that the run ID and single tool invocation are preserved. The default test uses SQLite as an explicit stand-in for the SQL storage path:

.venv/bin/python -m pytest tests/test_postgres_example.py -q

To run that same process-crash scenario against PostgreSQL, plus the existing backend tests, configure a test database whose role can create/drop isolated test schemas:

HARNESS_TEST_POSTGRES_DSN="$DATABASE_URL" .venv/bin/python -m pytest \
  tests/test_postgres_example.py tests/test_postgres_runtime.py \
  -k 'not missing_schema_privilege' -v

The excluded privilege test additionally needs permission to create roles. Tests use unique disposable schemas and clean up those schemas. The PostgreSQL crash test shortens its lease to two seconds; the CLI uses the normal 30-second default.

Local subprocess recovery and the live PostgreSQL integration suite are separate claims. Passing with a SQLite stand-in does not qualify PostgreSQL pooling, locking, schema provisioning, or your deployment's failover behavior. PostgreSQL remains subject to service-backed release qualification.