Orchestration Engine

The Orchestration Engine is the core agent execution runtime for AEGIS. It uses LangGraph to build a stateful graph pipeline that processes user messages through system prompts, memory retrieval, LLM calls, tool execution, skill injection, HITL approval checkpoints, and output formatting.

Overview

This service is the domain-neutral agent runtime at the center of AEGIS. Compliance domains are installable packs discovered at startup (packs/<name>/manifest.yaml, or out-of-tree via PACKS_DIR), and agent behavior — skills, routers, rules, prompts, settings — is tenant-configurable data served by the agent-config-service, not code in this repo. The five bundled packs (rule_37, rule_32, form_pr, flaring_monitor, epa_oooob) are all part of the first installed vertical (Texas RRC oil & gas / EPA); adding a domain is a new pack directory, not a core edit.

When a user sends a message to an AI agent, the orchestration engine:

1. Assembles the agent’s system prompt (persona + identity; for general mode, a router manifest sealed at conversation birth) and conversation history
2. Retrieves working memory and episodic memories
3. Calls the LLM (via LiteLLM) with budget enforcement
4. Executes any tool calls the LLM requests — including the R36 selection tools pull_routers / select_skill, which load skills mid-turn
5. Checks if HITL approval is required
6. Formats the final output

(The legacy marker-scanning skill selection hop was retired in R36, and the vestigial synthesis LLM phase was removed entirely in R41 B0 — the graph runs a single LLM node and approval_node flows straight to output_format.)

Beyond the core agent pipeline, the orchestration engine also hosts the compliance dashboard API, checklist management, rule lifecycle, and workspace SSE endpoints.

Port & Language

Key Endpoints

Authorization (R42d)

All user-facing routers require perimeter-stamped identity via aegis_shared.auth (get_current_user — including /execute and the SSE stream), mode-gated by AEGIS_AUTH_MODE. Fine-grained gates on top: filings.submit on checklist submit, filings.review on checklist reject, eval.read / eval.run on the eval routes, power_user/admin on rule-version writes and /compliance/assess, and skills.author on the sandbox code-block dry-run. See Auth Flow — Service-Side Enforcement.

Agent Execution

Conversations

Compliance Dashboard (R1.2)

Checklists (R1.3)

Artifacts

Rules (R1.5)

Workspaces (R1.4)

Eval Capture (R40a)

At turn end (both the SSE turn runner and the sync /execute path) the full assembled context publishes fire-and-forget to the snapshots:pending Redis stream; a background consumer-group writer drains it into turn_snapshots (ack-after-persist, so entries survive restarts). See the database schema page for the tables.

Architecture

LangGraph Pipeline

The core execution follows a StateGraph (R36 — the skill_select_node / skill_inject_node hop is deleted; skill selection happens inside the tool loop):

START -> system_prompt_node -> memory_node -> resume_guard -> initial_llm_call
      -> [tool_node -> initial_llm_call]* (tool loop; pull_routers /
         select_skill execute here as ordinary tool calls)
      -> approval_node -> output_format -> END

Routing logic between nodes:

• After initial_llm_call: If there are pending tool calls, route to tool_node. If there is an error, route to output_format. Otherwise, route to approval_node.
• After tool_node: If a HITL gate fired (awaiting_hitl), route to output_format. Check iteration limit. Otherwise route back to initial_llm_call.
• approval_node flows straight to output_format (R41 B0 removed the vestigial synthesis phase). It either pauses awaiting_hitl or passes through — either way the next stop is output_format.

Module Breakdown

src/orchestration/
├── main.py                     # FastAPI app, /execute and /stream endpoints
├── engine.py                   # LangGraph StateGraph construction and compilation
├── nodes.py                    # Node implementations (1 function per node)
├── tools.py                    # Tool definitions and execution for LLM function calling
│                               #   (hosts SKILL_SELECT_MARKER_TOMBSTONE — the retired legacy selection marker)
├── selection_tools.py          # R36: pull_routers / select_skill tool handlers
├── conversation_runtime.py     # R36: sealed-prefix runtime state (conversation_runtime_state)
├── context_maintenance.py      # R36: per-turn router deltas, lazy upgrades, tombstones
├── state.py                    # GraphState TypedDict (flows through the graph)
├── schemas.py                  # Request/response Pydantic models
├── services.py                 # HTTP clients for memory, KG, approval services
├── sandbox/                    # Unix-socket client + pipeline for the sandbox-runner
│   ├── python_client.py        #   (R33): ships skill code blocks to the jailed runner
│   ├── pipeline.py             #   AST pre-validation → socket → structured result
│   └── harness.py              #   fail-closed governance gate around execution
├── budget.py                   # Token/cost budget enforcement
├── config.py                   # Settings from environment variables
├── conversation_routes.py      # Conversation CRUD (list, create, get, update, delete)
├── packs/                      # Pack discovery (R41 B2): manifest per compliance domain
│   ├── __init__.py             # discover_packs() / load_packs()
│   ├── schema.py               # PackManifest schema (v1)
│   ├── rule_37/manifest.yaml   # Skills + checklist template for Rule 37
│   ├── rule_32/manifest.yaml   # … Rule 32
│   ├── form_pr/manifest.yaml   # … Form PR
│   ├── flaring_monitor/manifest.yaml
│   └── epa_oooob/manifest.yaml # … EPA OOOOb (fifth domain, platform-expansion)
├── skills/                     # Checklist-driven skill implementations
│   ├── __init__.py             # Slug-keyed skill registry (register_skill, list_registered_skills)
│   ├── base.py                 # Base skill class (SKILL_KEY identity)
│   ├── declarative.py          # Descriptor-driven skills (R41 B4): manifest descriptors, no Python
│   ├── rule37/                 # Rule 37 spacing exception skills (11 skills)
│   ├── rule32/                 # Rule 32 flaring exception skills (10 skills)
│   ├── form_pr/                # Form PR production report skills (8 skills)
│   ├── flaring_monitor/        # Flaring monitor skills (6 skills)
│   └── field_event/            # Field event conversation skills
├── compliance/                 # Compliance dashboard and workspace
│   ├── routes.py               # /compliance/summary, /matrix, /charts
│   ├── engine.py               # Compliance status computation engine
│   ├── queries.py              # Cypher query templates for compliance
│   ├── checklist_routes.py     # Checklist CRUD routes
│   ├── artifact_routes.py      # Artifact management routes
│   ├── rule_routes.py          # Rule version CRUD routes
│   ├── rule_lifecycle.py       # Immutable rule versioning logic
│   ├── rule_monitor/           # Rule change monitoring
│   └── workspace/              # Entity workspace (assessment, SSE, agent)
│       ├── routes.py           # Workspace API routes
│       ├── assessment.py       # Automated assessment logic
│       └── events.py           # WorkspaceEventType enum, SSE event formatting
├── seed_checklists.py          # Checklist template seeder (content from pack manifests)
├── seed_rules.py               # Compliance rule data seeding script
├── seed_conversations.py       # Demo conversation seeding
└── seed_demo_data.py           # Combined demo data seeder

GraphState

The GraphState TypedDict carries all execution state through the graph:

class GraphState(TypedDict, total=False):
    execution_id: str
    conversation_id: str
    agent_id: str
    agent_type: str              # "rule_37", "rule_32", etc.
    tenant_id: str
    status: str                  # "pending", "running", "completed", "awaiting_hitl", "failed"
    messages: list[dict]         # Conversation messages (role/content/tool_calls)
    system_prompt: str
    model: str                   # LLM model to use (default: gpt-4o)
    loaded_skills: list          # skill_keys currently loaded (R36: grows mid-turn via select_skill)
    injected_skill_ids: list     # Skills already injected
    active_skills: dict          # skill_id -> tier level
    tokens_used: int
    cost_usd: float
    max_tokens: int
    max_cost_usd: float
    pending_tool_calls: list     # Tool calls awaiting execution
    iteration: int               # Guards against infinite loops
    memory_context: dict         # Working + episodic memory data
    hitl_required: bool
    hitl_checkpoint_type: str
    hitl_approval_id: str | None

Budget Enforcement

The budget.py module enforces two independent tiers (R41 A8):

• Per-conversation (cumulative across every turn): BudgetExceededError when tokens_used >= max_tokens or cost_usd >= max_cost_usd.
• Per-turn (spend since the turn began): BudgetExceededError when tokens_used - turn_tokens_start >= max_tokens_per_turn (or the cost equivalent) — bounds a single runaway turn while the conversation still has headroom.

tokens_remaining returns the tighter of the two tiers, so the per-call output ceiling never blows past either. Budget is checked before every LLM call. When exceeded, /execute returns HTTP 429 and the SSE stream emits an error event. Per-turn env defaults (DEFAULT_MAX_TOKENS_PER_TURN / DEFAULT_MAX_COST_PER_TURN) equal the per-conversation ceiling, so the per-turn tier is dormant until an operator lowers it.

Platform Settings Resolution (Tenant LLM Config)

At every turn start — both the synchronous /execute path and the SSE stream — the engine fetches GET /settings from the agent-config-service (with X-Tenant-Id/X-User-Id headers) and resolves the model and per-execution budgets as:

request override → tenant platform setting → env default

The env defaults (DEFAULT_LLM_MODEL, DEFAULT_MAX_TOKENS_PER_EXECUTION, DEFAULT_MAX_COST_PER_EXECUTION, DEFAULT_MAX_OUTPUT_TOKENS_PER_CALL, DEFAULT_THINKING_MODE) remain the fallback floor; a settings fetch failure degrades to them.

The last two are per-LLM-call knobs (no request override): max_output_tokens_per_call is the max_tokens sent on each call — thinking + visible output share it, so on Claude Sonnet 5 (adaptive thinking by default) a low ceiling can be consumed entirely by a thinking block and yield an empty assistant message; llm_call now fails the turn loudly when that happens (finish_reason=length with no content and no tool calls) instead of appending an empty message. thinking_mode (adaptive | disabled) is sent explicitly to Anthropic models (omitted for Haiku-tier/non-Anthropic, and re-derived when a call retries on the fallback_model). Admins manage the tenant values from the Platform Settings page. If the tenant sets a fallback_model, every LLM call site retries once on that model when the primary model errors. Seeded agent_definitions no longer pin model_config.model_preference (now null), so the tenant default_model governs — an explicitly set per-agent model_preference still overrides. Workspace assessment skills and the scoped agent use the same settings.

Daily caps (tenant-aggregate plus optional per-user overrides) are enforced with Redis counters — usage:daily:{tenant}:{yyyymmdd}:tokens|cost and usage:daily:{tenant}:{user}:{yyyymmdd}:* (48h TTL). The counters are checked before a turn starts (429 on /execute; a terminal SSE error event on the stream) and incremented with actual usage after each turn — so an in-flight turn can overshoot a cap, and it is the next turn that gets blocked.

Skill Selection (R36 — message-driven)

The legacy marker-scanning selection flow is gone. General-mode runs (an agent_definitions row with root_skill_key = NULL) receive a router manifest (router_key + recruitment_blurb per active router) in the index-0 system prompt block, plus two selection tools (orchestration/selection_tools.py):

1. pull_routers(router_keys[]) — returns the pulled routers’ selectable skills with full descriptions (derived read over approved mappings × active skills × latest-approved description version)
2. select_skill(skill_key) — fires the existing load_skill() machinery: context templates land as system messages, the skill’s llm_visible code-block tools join the next tool assembly

Validation failures return typed errors and fail toward deny — domain tools do not exist in the run until select_skill succeeds. Agent-mode runs (rule_37, rule_32) load their root skill at conversation start and do not pull routers.

The general-mode index-0 block is sealed at conversation birth (conversation_runtime_state, migration 016) and reused byte-for-byte on reopen; router changes, lazy skill version upgrades, and skill-deletion tombstones reach live conversations as tail appends only (conversation_context_appends), applied per-turn by context_maintenance.py.

HITL Approval Checkpoints

HITL fires from the seeded require_hitl SkillRules on the RRC skills (R35 P5): a rule matches the executed tool name on after_tool_call (e.g. rule37_filing_assembly → pre_filing, good_cause_narrative → good_cause_review, rule32_filing_assembly → pre_filing). A before_tool_call platform rule gates mutating capabilities, and request metadata can still override hitl_checkpoint_type.

When HITL is required, the engine creates an approval request via the approval service and pauses execution with status: "awaiting_hitl".

Tool Definitions

Since R35 P4 the visible tool list derives from provenance, not agent type: core capabilities (entity_resolve, context_assemble, and since R43 render_chart) are always present; R36 adds the selection tools (pull_routers, select_skill) for general mode; everything domain-specific arrives from the loaded skills’ llm_visible code blocks. Those code blocks are not run in-process — they are shipped over a unix socket to the sandbox-runner (nsjail-jailed Python), which is what keeps tenant-authored skill code from touching the host:

R38 — the Skill Creator’s authoring tools. The platform creator agent (root skill skill_creator, seeded by agent-config-service) gets ten tools (R39 P0 added the list_categories read) from orchestration/authoring_tools.py — native handlers, not sandboxed code blocks — but visibility stays data-driven: their schemas are unioned into the LLM payload only when the skill_creator skill is loaded (no agent_type branch). All six write tools are in MUTATING_TOOLS (fail-closed when governance is unevaluable), and they act as the conversing user via fail-closed X-User-Id headers, so owner_id lands as the real user. The Creator authors only — it has no approve/map/retire/invoke tools; admin approval (publish-on-approve, R38) is what makes an authored skill loadable and reachable.

The deferred flaring_monitor / compliance_monitor agents keep a hardcoded fence set (R35-FENCE) until their skills land. See LangGraph Pipeline for the full breakdown.

Sandbox Runner (Jailed Code Execution)

Skill code blocks are tenant-authored, so they never execute in the orchestration process. The sandbox-runner (services/sandbox-runner/) is the platform’s only privileged component: a stdlib-only root server, reachable solely over a local unix socket (root:aegis, 0660 — never a TCP port), that spawns nsjail to run a single Python code block under namespaces + seccomp + cgroups and returns a structured JSON result. It has no service page of its own because it has no HTTP API.

The defense is two-layer: an unprivileged AST validator (aegis_shared.sandbox.validation) is the cheap first pass, and the deny-by-default seccomp policy (infrastructure/deploy/sandbox/python.seccomp.kafel) is the load-bearing containment. The orchestration side lives in src/orchestration/sandbox/ (python_client.py → socket, pipeline.py, harness.py); the fail-closed governance gate blocks classification-bearing code when governance is unevaluable (R41 Phase A). Socket path and limits are configured via AEGIS_SANDBOX_SOCKET, AEGIS_SANDBOX_WALL_S, and AEGIS_SANDBOX_MAX_OUTPUT_BYTES.

Dependencies

Python Packages

Service Dependencies

Configuration

Replay bind (R40b eval)

The eval runner drives cases through a separate FastAPI app (orchestration.replay_app:replay_app) bound loopback-only on 127.0.0.1:8101, run as its own systemd unit (aegis-replay.service). It is not a second bind of the main app — that would expose /replay publicly via Caddy’s forward_auth. It mounts only POST /replay (plus /health) and shares the engine internals by import; the replay_origin flag exists only on its request model, so there is no public surface to forge it on. Nothing in the Caddyfile references the port → it is structurally unreachable from the edge.

Observability (Langfuse Tracing)

The engine ships optional Langfuse  tracing, wired in orchestration/observability.py. It is off by default and activates only when both LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set — otherwise every tracing helper is a no-op and the service behaves identically.

How it works:

• At startup (init_tracing() in the FastAPI lifespan) the engine enables the LiteLLM langfuse_otel callback for all litellm.acompletion calls and initialises the Langfuse SDK (v4, OpenTelemetry-based).
• Each agent run is wrapped in a parent span via trace_agent_run(...). LiteLLM generation spans nest under it automatically: both layers ride the global OTEL TracerProvider and share OTEL context, and Langfuse’s span processor exports gen_ai.* spans by default — yielding one trace per agent run with each LLM call as a nested generation (model, tokens and cost captured automatically).

Trace attributes set:

Instrumented entry points: POST /execute, GET /conversations/{id}/stream, workspace assessment (GET /workspaces/{id}/stream), and the scoped agent (POST /workspaces/{id}/items/{idx}/agent).

R34 entity-layer child spans (orchestration/entity_spans.py): the entity resolution / context-assembly / HITL pipelines emit child_span observations that nest under the agent-run span, so an auto-select decision and the full HITL flow are reconstructable from the trace alone. They carry the internal, precise values (the audit audience, not the LLM) and are no-ops when tracing is off. All fire orchestration-side — the knowledge-graph service is untraced and OTEL context doesn’t cross the HTTP boundary, so KG returns the facts (the resolution_trace block on the resolve response) and the orchestration tool wrappers + gate/guard emit the spans.

The entity_resolve.outcome span is a separate layer from the platform:log_entity_resolution rules-audit entry — the spans only wrap the resolve callsite and never re-fire the on_entity_resolve trigger (one audit entry + one outcome span per auto-select). An approved-but-unexecuted mutation is the absence of a hitl.resume_replay span next to an approved approval record. See docs/specs/phases/R34-phase-6-calibration-and-demo-runbook.md.

Running Locally

cd services/orchestration-engine
poetry install
poetry run uvicorn orchestration.main:app --reload --port 8001

Seeding Data

The service includes several seeding scripts:

# Seed checklist templates
poetry run python -m orchestration.seed_checklists
 
# Seed compliance rules
poetry run python -m orchestration.seed_rules
 
# Seed demo conversations
poetry run python -m orchestration.seed_conversations