Decisions with Jev

Use the standalone decision SDK to route a request, classify a document, or review an answer. It wraps TypeSafe Jev's Choice, Score, and Noul primitives in typed HarnessX requests and results.

Import it from harnessx.decisions. Decisions have their own async provider and usage accounting. An application calls the provider before selecting an agent or after receiving an answer; automatic checks inside the agent loop remain planned.

Choose a question type

HarnessX question	Jev primitive	Result	Used in the examples
`ChoiceQuestion`	`Choice`	Selected label, probabilities for every option, and confidence	Agent routing and document classification
`ScoreQuestion`	`Score`	Unrounded expected score, ordered probabilities, rubric levels, and confidence	Answer coverage
`BooleanQuestion`	`Noul`	Probability that the proposition is true	Support from supplied evidence

Choice requires at least two named options with descriptions. Score accepts two to ten ordered descriptions, indexed from zero. A four-level rubric produces a score between 0 and 3, which may be fractional.

BooleanCriteria optionally describes the true and false outcomes. Either description can be omitted. Noul returns a probability, without a separate confidence value. Compare answer.probability with a threshold chosen for your application; bool(answer) raises to prevent implicit conversion.

Try the offline examples

From a clone of the repository (uv sync --all-extras), run:

uv run python -m examples.jev_routing --min-confidence 0.8
uv run python -m examples.jev_classification
uv run python -m examples.jev_answer_review

These use fixed decisions and scripted agent responses. They need no Jev extra, API keys, or external services. Outputs demonstrate integration behavior; they do not classify changed inputs or measure model quality. The examples are source-checkout entry points and are not bundled in the SDK wheel.

The same three examples appear in the harness-web example library as offline demos.

Install and enable live decisions

Install the optional extra and configure a TypeSafe key:

uv add "harnessx[jev]"
export TYPESAFE_API_KEY='your-key'

The jev extra installs typesafe-sdk>=0.7.1,<0.8; it is also included in all. Importing harnessx or harnessx.decisions does not load the vendor SDK. Constructing JevDecisionProvider requires the extra.

Add --live explicitly to call Jev:

uv run python -m examples.jev_classification --live --text 'Document text here'
uv run python -m examples.jev_answer_review --live
uv run python -m examples.jev_routing --live --min-confidence 0.8 \
  --provider anthropic --model YOUR_MODEL_ID

Live classification and review require the Jev extra and TYPESAFE_API_KEY. Live answer review still uses a scripted agent answer, then sends that answer and its evidence to Jev. Live routing also calls the selected generative model: supply an explicit model ID, its provider credentials, and its optional extra where required. Live calls may incur service charges.

Evaluate typed questions

This complete example sends a query, answer, and evidence in one request. Each question independently assesses that supplied state. It makes a live Jev call when run.

import asyncio
from harnessx.decisions import (
    BooleanCriteria,
    BooleanQuestion,
    ChoiceQuestion,
    DecisionRequest,
    JevDecisionProvider,
    ScoreQuestion,
)

async def main():
    request = DecisionRequest(
        check_id="answer_review",
        version="v1",
        state={
            "query": "What were sales in January and February?",
            "answer": "Sales were $100 in January and $120 in February.",
            "evidence": "Sales report: January $100; February $120.",
        },
        questions={
            "route": ChoiceQuestion(
                instructions="Choose the best route for the query.",
                options={
                    "sql": "Internal structured data or SQL.",
                    "research": "External research and source synthesis.",
                    "general": "General writing and explanations.",
                    "unknown": "Insufficient context or no clear route.",
                },
            ),
            "coverage": ScoreQuestion(
                instructions="How completely does the answer address the query?",
                levels=(
                    "Does not answer the request.",
                    "Addresses a small part, with major omissions.",
                    "Addresses most of the request, with minor omissions.",
                    "Fully addresses every part of the request.",
                ),
            ),
            "supported": BooleanQuestion(
                instructions="Does the evidence support every factual claim?",
                criteria=BooleanCriteria(
                    true="Every claim follows from the supplied evidence.",
                    false="A claim is unsupported or contradicts the evidence.",
                ),
            ),
        },
    )

    async with JevDecisionProvider() as decisions:
        batch = await decisions.evaluate(request)

    if batch.status == "unavailable":
        assert batch.error is not None
        print(batch.error.code, batch.error.message)
        return

    print("Route:", batch.choices["route"].choice)
    print("Coverage:", batch.scores["coverage"].score)
    print("Evidence support:", batch.booleans["supported"].probability)

asyncio.run(main())

The adapter translates those question objects to the official SDK's Choice, Score, and Noul, then calls AsyncTypeSafeClient.system_one(). It preserves the resolved model and available service request ID in the result. The default endpoint is the direct TypeSafe API; OpenRouter compatibility for this adapter has not been verified.

Route to one agent

The routing example source uses Choice to select from an application-defined mapping of agent factories:

Decision	Application behavior
`sql`, `research`, or `general`, with sufficient confidence	Construct and run that one agent
`unknown` or an unexpected label	Use the general agent
Missing confidence or confidence below `min_confidence`	Use the general agent
Unavailable decision	Use the general agent
Selected agent fails	Return its failed result; do not start a second agent

min_confidence is required and must be a finite number between zero and one. The example command's 0.8 is illustrative, not a calibrated default. The model selects a label; application code owns the factories and configuration. The minimal specialists draft answers; the example does not connect a database or browse sources.

Classify documents and review answers

The classification example source asks a Choice question with five labels: W2, Deposit, Insurance, Payroll, and unknown. It returns the typed decision and its full distribution. The offline fixture always returns W2; pass --live --text ... to assess supplied document text.

The answer-review example source makes a separate decision request after a successful Agent.run() result:

Score evaluates coverage against four ordered levels, from unanswered to fully addressed.
Noul evaluates whether every factual claim follows from the supplied evidence, using explicit true and false criteria.

The review skips unsuccessful runs and leaves the original run's status, answer, and generative-model usage unchanged. It reports probabilities and scores; it does not turn them into an automatic pass/fail gate. Jev assesses the evidence you supply and does not retrieve or independently verify sources.

Handle results and failures

DecisionBatch.status is either completed or unavailable. A completed batch requires every requested answer to validate. An uncertain but valid answer remains completed; confidence thresholds belong to the application.

Missing or incompatible answers, mismatched option labels or rubric levels, nonfinite probabilities, invalid distributions, and inconsistent scores produce an unavailable batch with empty answer maps. The adapter ignores extra answer IDs after SDK decoding. Probability distributions must sum to one within an absolute tolerance of 1e-3; they are never silently renormalized. Scores must agree with their distribution's weighted mean within 1e-3 * max(1, top_level).

Condition	Outcome
Missing optional dependency	`ImportError` with installation guidance
Invalid local configuration, question, or oversized state	Local exception before I/O
Authentication, authorization, or request rejection	Unavailable, without retry
Connection failure, transport timeout, HTTP 429 or 500/502/503/504	Bounded retry, then unavailable if unresolved
Invalid or missing requested answers	Unavailable, without retry
Caller cancellation	`asyncio.CancelledError` propagates
Programming error	Exception propagates

DecisionError exposes a stable code, sanitized message, optional HTTP status, and whether the cause is retryable. Codes are authentication, forbidden, invalid_request, rate_limited, timeout, connection, server, invalid_response, and unexpected_http. Raw server bodies are excluded from these diagnostics. The upstream SDK has its own logging configuration; debug logging can include bodies.

Configure limits and ownership

Provider option	Default	Meaning
`model`	`jev-1.13.0`	Explicit model sent on every request
`timeout_seconds`	`2.0`	Total evaluation deadline, including retry delays
`max_retries`	`1`	At most two HTTP attempts by default
`max_state_bytes`	`16384`	Limit on compact UTF-8 JSON state
`api_key`	`TYPESAFE_API_KEY`	Credential used when constructing a client
`base_url`	`https://api.typesafe.ai`	Explicit API root; does not inherit `TYPESAFE_BASE_URL`
`client`	`None`	Optional borrowed `AsyncTypeSafeClient`

These are configurable engineering limits, not measured Jev latency claims. evaluate(request, timeout_seconds=..., max_attempts=...) can shorten the configured deadline and cap attempts. SDK retries are disabled so they cannot multiply the adapter's retry budget. A Retry-After delay is honored when it fits the remaining deadline; otherwise the evaluation returns unavailable.

Use async with JevDecisionProvider(...) to close resources reliably. Clients created by the adapter are owned and closed by it. Injected clients remain caller-owned; combining client with api_key or base_url raises. Per-call overrides do not change a borrowed client's defaults.

The provider supports concurrent evaluations. aclose() is idempotent, cancels outstanding evaluations, and closes owned resources. Evaluating after close raises RuntimeError.

Preserve state, usage, and results

Request state must be text, a JSON object, or a JSON array containing only finite JSON values and string object keys. State is copied at construction and again before dispatch. Arbitrary objects are never converted with str().

Requests and results expose to_dict() and from_dict() with schema_version=1. Decode validates the data, rejects unknown schema versions and question/answer kinds, and accepts additive metadata fields. Use those codecs before json.dumps; score distributions become arrays, and grouped answer maps are read-only.

A batch preserves check_id, version, requested and resolved model, available request ID, elapsed time, and attempt count. Token counts describe the successful response. After retries, earlier attempts may have consumed unreported tokens, so usage.incomplete remains true. Absent measurements stay None, including cost: the adapter does not estimate a dollar figure. Decision usage is separate from agent model usage.

Implementation status

The standalone SDK, all three examples, and offline harness-web entries are implemented. The adapter has tests through the real TypeSafe SDK with a mocked HTTP transport, plus cancellation, lifecycle, validation, packaging, and offline example checks. Those checks do not establish live service access or decision quality.

Automatic agent-loop observation, persisted runtime assessments, approval enforcement, and automatic interventions remain planned. See Implementation status for the broader SDK boundaries and API reference for the public contracts.