Eval Capture

Expert verdict capture (R40a). A tester marks a completed assistant turn good or bad in one gesture; the verdict persists as a replayable eval case in Postgres (the system of record) with a frozen context snapshot, and the score mirrors best-effort to the Langfuse trace. All routes proxy to the orchestration engine and require authentication — the gateway injects X-User-Id / X-Tenant-Id, which are the only identity source (never the body).

Endpoints

POST /api/v1/eval/captures

Upserts on (execution_id, captured_by) — a second gesture from the same user updates the existing case (score flips take the new value; enrichment fields are sticky and survive a later bare gesture). The eval_cases insert is the only operation whose failure fails the request; snapshot linkage and the Langfuse mirror degrade explicitly via snapshot_status / mirror_status.

Request body:

{
  "execution_id": "a1b2c3d4-...",
  "conversation_id": "conv-3f2a9b1c44d0",
  "score": -1,
  "expected_output": "The agent should have cited 16 TAC 3.37(h)(2).",
  "note": "Missed the exception citation",
  "category_tag": "citation"
}

score must be -1 or 1. expected_output, note, and category_tag are optional — a bare thumbs gesture is a valid score-only case.

Response:

{
  "id": "6f0e...",
  "execution_id": "a1b2c3d4-...",
  "score": -1,
  "snapshot_status": "linked",
  "mirror_status": "mirrored",
  "replay_mode": "frozen"
}

snapshot_status: "missing" means the turn’s context snapshot had not landed when the capture was written (checked once, retried once after ~2s); a boot-time re-link pass upgrades it when the snapshot appears. replay_mode (frozen when linked, live otherwise) tells the R40b runner how the case can be replayed.

GET /api/v1/eval/captures

Query parameter conversation_id (required). Returns only the requesting user’s captures — the UI uses this to render captured state (filled thumbs) on messages reloaded from the database.

{
  "captures": [
    {
      "execution_id": "a1b2c3d4-...",
      "score": -1,
      "expected_output": "…",
      "note": "…",
      "category_tag": "citation",
      "snapshot_status": "linked",
      "mirror_status": "mirrored"
    }
  ],
  "total": 1
}

GET /api/v1/eval/metrics

Internal counts for the missing-rate bug detector (steady-state missing is zero outside a fast-click within seconds of turn completion; nonzero means the snapshot writer is broken):

{ "total": 42, "missing": 0, "mirror_failed": 1, "mirror_pending": 0 }

Eval Runs (R40b)

The eval runner drives cases (Creator pins + expert-corrected cases) through the current agent code and records a baseline-able run. This is live-drive replay: the turn’s input is reconstructed and re-executed against today’s skill loading / entity resolution / model routing — never a frozen re-issue of the old messages. Grading is judge-vs-expected (expert cases) or shape-assertion (pins), never byte-identity.

The runner is an on-box CLI that drives a loopback-only replay bind (replay_app, 127.0.0.1:8101) — it is NOT reachable through the gateway (§3, by design). The dashboard reads these Postgres-backed endpoints (never Langfuse):

Running the eval

poetry run python -m orchestration.eval.runner \
  --base-url http://127.0.0.1:8101 \
  --corpus pins,expert \
  --tag baseline-sonnet5-r40b [--case <id>] [--dry-run]

Per case: seed the frozen prefix (multi-turn only, via the cold-resume path) → POST /replay (replay_origin=true) → assert the replay’s trace has generations attached (retry ~3×/30s; distinguishes ingestion lag from the zero-generations litellm bug) → grade → record. --dry-run grades and prints but records nothing.

Baselines & regressions

A baseline is a tagged, completed run flagged is_baseline — at most one per (tenant, model). GET …/regressions auto-resolves the baseline for the run’s model_id and returns the pass→fail flips. If no baseline is promoted for that model it returns 409 (no baseline for model <model_id>) — a loud refusal, never a silent cross-model diff. Promotion is an explicit operator action (a run never becomes a baseline by being newest); it demotes the incumbent and promotes the target in one transaction (§8c).

GET /api/v1/eval/runs/{id} response:

{
  "run": {
    "id": "…", "tag": "baseline-sonnet5-r40b", "model_id": "anthropic/claude-sonnet-5",
    "git_sha": "555705c3…", "status": "completed", "is_baseline": true,
    "judge_threshold": 0.7, "n_cases": 6, "n_pass": 5, "n_fail": 1, "n_degraded": 0,
    "tokens_used": 48213, "cost_usd": 0.42
  },
  "case_results": [
    { "case_source": "pin", "case_ref": "P-A1", "grader": "pin_check",
      "verdict": "pass", "judge_score": null, "violations": [] },
    { "case_source": "expert", "case_ref": "6f0e…", "grader": "llm_judge",
      "verdict": "fail", "judge_score": 0.4, "violations": { "rationale": "…" } }
  ]
}

Verdicts: pass / fail (judge score vs judge_threshold, or an empty pin violation list) and degraded — a graded case whose health failed (gen-attach assertion, a judge call error, or a vacuous/agent-less pre-flight), distinct from a content fail. Verdicts are advisory: no CI gate, no deploy block.

Judge model: the expert grader passes an explicit model= (EVAL_JUDGE_MODEL, default anthropic/claude-sonnet-4-6) so it does not self-grade with the model under test, and runs in its own trace so its generations don’t pollute the conversation trace.