Execution Engine
The Execution Engine is the component that runs validated Plans. It transforms a static Plan IR document into a live, governed, observable process — invoking capabilities sequentially, managing data flow, enforcing policies per-step, handling gates and interaction points, recording traces, and managing the outbox for external effects. The engine is a durable state machine with persistent checkpointing: plan executions survive process restarts, transient failures are retried automatically, and external effects are deferred until all steps succeed. This specification defines its architecture, execution lifecycle, durability model, error handling, and integration with every other Zakardo subsystem.
1. Design Goals
Section titled “1. Design Goals”- Execute what the plan says. The Execution Engine does not reason about goals or choose alternatives. It runs the validated Plan IR faithfully. All reasoning happened in the Planner; all correctness was verified by the Validator.
- Sequential execution in v1. Steps execute one at a time, in order. Pauses (gates, interactions, retries) block the entire plan. This is intentionally simple. Parallel execution is a future enhancement.
- Durable by default. Every state transition is checkpointed to persistent storage. Process restarts do not lose executions. Paused plans survive indefinitely. Retrying plans pick up where they left off.
- Transient failures are invisible to users. Network blip? Service momentarily down? The engine waits and retries. The user sees “retrying step 3…” not “plan failed.”
- Every step is governed. Per-step policy evaluation means that even if a plan passed validation, runtime conditions (rate limits, dynamic constraints) can still gate or block individual steps.
- Every step is observed. Every capability invocation, gate resolution, interaction response, error, retry, and timing is recorded in the Trace. No silent execution.
- Execution modes are real. SIMULATED, DRY_RUN, LIVE, and TRAINING are not labels — they fundamentally change what the engine does. Services are mode-resolved, outbox behavior changes, and capabilities may receive entirely different implementations.
- External effects are deferred. The Outbox stages external side effects during execution but only commits them after all steps succeed. If any step permanently fails, no external effects are realized.
- Errors are structured and categorized. Transient vs. permanent failure distinction drives automatic behavior: retry the transient, stop on the permanent.
2. Architecture Overview
Section titled “2. Architecture Overview”2.1 The Durable State Machine
Section titled “2.1 The Durable State Machine”The execution engine manages each plan execution as a persistent state machine. Instead of a single async function that runs to completion (and dies if the process restarts), the engine:
- Advances the state machine one transition at a time.
- Checkpoints the full execution state to persistent storage after every transition.
- Can be woken up by triggers: timers (retry), external signals (gate response, interaction response), or process recovery.
This means:
- A plan paused at a gate survives server restarts.
- A plan waiting to retry a transient failure resumes automatically.
- No execution data is lost unless the checkpoint store itself is lost.
2.2 Execution State Machine
Section titled “2.2 Execution State Machine”INITIALIZED │ ▼EXECUTING_STEP ◄──────────────────┐ │ │ ├── step succeeds ─────────────┤ (checkpoint, advance to next step) │ │ ├── transient failure ───► WAITING_RETRY │ │ (backoff timer) │ │ timer expires → re-execute step │ │ ├── gate triggered ────────► WAITING_GATE │ │ (user responds → resume) │ │ ├── interaction required ──► WAITING_INTERACTION │ │ (user responds → resume) │ │ └── permanent failure ─────► PERMANENTLY_FAILED │ ▼ (discard staged outbox, finalize trace)
All steps completed ──────────► COMMITTING_OUTBOX │ ├── outbox commit succeeds ──► COMPLETED │ └── outbox commit fails ──► COMMITTING_OUTBOX (retry with idempotency keys)Terminal states: COMPLETED, PERMANENTLY_FAILED
Waiting states: WAITING_RETRY, WAITING_GATE, WAITING_INTERACTION — the plan is alive but paused. The checkpoint is persisted. A trigger will resume it.
2.3 What Triggers State Transitions
Section titled “2.3 What Triggers State Transitions”| Trigger | Source | Wakes State |
|---|---|---|
| New execution submitted | HTTP request from Runtime API | Creates INITIALIZED, immediately advances to EXECUTING_STEP |
| Step completed | Internal (engine loop) | EXECUTING_STEP → EXECUTING_STEP (next step) or COMMITTING_OUTBOX |
| Retry timer expires | Scheduled timer (setTimeout in-process, or external scheduler) | WAITING_RETRY → EXECUTING_STEP |
| Gate response received | HTTP/WebSocket from frontend | WAITING_GATE → EXECUTING_STEP |
| Interaction response received | HTTP/WebSocket from frontend | WAITING_INTERACTION → EXECUTING_STEP |
| Process recovery | Startup recovery sweep | Any non-terminal state → re-evaluate and advance |
| Outbox commit completed | Internal | COMMITTING_OUTBOX → COMPLETED |
3. Execution Engine Interface
Section titled “3. Execution Engine Interface”3.1 Input
Section titled “3.1 Input”interface ExecutionRequest { plan: PlanIR; // Validated Full Plan IR validation_id: string; // Proof of validation user_context: UserContext; // Who is executing execution_mode: ExecutionMode; // LIVE, DRY_RUN, SIMULATED, TRAINING environment: string; // 'production', 'staging', 'development' intent_inputs: Record<string, any>; // User-provided inputs that satisfy plan.input_schema options?: ExecutionOptions;}
interface ExecutionOptions { timeout_ms?: number; // Overall execution timeout (default: 300000 = 5 min) step_timeout_ms?: number; // Per-step timeout (default: 30000 = 30s) gate_timeout_ms?: number; // How long to wait for gate responses (default: 3600000 = 1 hour) interaction_timeout_ms?: number; // How long to wait for interaction responses (default: 3600000 = 1 hour) retry_budget_ms?: number; // Max total time spent in WAITING_RETRY across all steps (default: 300000 = 5 min) trace_level?: 'minimal' | 'standard' | 'verbose'; // Trace detail level}
type ExecutionMode = 'LIVE' | 'DRY_RUN' | 'SIMULATED' | 'TRAINING';3.2 Output
Section titled “3.2 Output”The Execution Engine emits a stream of events and produces a final result when execution reaches a terminal state.
// Final result (available after execution reaches terminal state)interface ExecutionResult { execution_id: string; plan_id: string; plan_version: string; status: ExecutionStatus; execution_mode: ExecutionMode; outputs: Record<string, any>; // Plan-level outputs (from output_schema mapping) trace_id: string; // Reference to the full Trace started_at: string; completed_at: string; duration_ms: number; // Wall clock time from start to finish active_duration_ms: number; // Time spent actually executing (excludes waits) step_results: StepResult[]; dry_run_report?: DryRunReport; // Only for DRY_RUN mode error?: ExecutionError; // If status is PERMANENTLY_FAILED}
type ExecutionStatus = | 'COMPLETED' // All steps succeeded, outbox committed | 'PERMANENTLY_FAILED' // A step permanently failed | 'WAITING_RETRY' // Paused, waiting to retry a transient failure | 'WAITING_GATE' // Paused at a governance gate | 'WAITING_INTERACTION' // Paused at an interaction point | 'COMMITTING_OUTBOX'; // All steps done, committing external effects
interface StepResult { step_index: number; step_name: string; capability: string; capability_version: string; status: StepStatus; started_at: string; completed_at?: string; duration_ms?: number; outputs?: Record<string, any>; error?: StepError; gate_resolution?: GateResolution; interaction_response?: InteractionResponse; policy_evaluation?: PolicyEvalResult; retry_count: number;}
type StepStatus = | 'PENDING' // Not yet started | 'COMPLETED' // Finished successfully | 'FAILED' // Permanently failed (retries exhausted or non-retryable) | 'RETRYING' // Failed transiently, will retry | 'GATED' // Paused at a gate | 'INTERACTING' // Paused at an interaction point | 'ABORTED'; // User denied gate or aborted interaction3.3 Event Stream
Section titled “3.3 Event Stream”During execution, the engine produces events for every significant state transition. Events follow a store-then-emit pattern: every event is first persisted to the Event Store (see §4.5), then emitted to the Event Channel (see §4.6) for any connected clients. This guarantees that:
- Events are never lost, even if no client is connected.
- Clients can reconnect and catch up by replaying stored events.
- The Event Store is the source of truth; the live channel is a convenience.
Every event carries a monotonically increasing event_id within its execution scope:
interface ExecutionEventBase { event_id: string; // Monotonic ID within this execution (e.g. "evt_001") execution_id: string; timestamp: string; // ISO 8601}
type ExecutionEvent = | (ExecutionEventBase & { type: 'EXECUTION_STARTED'; plan_id: string; total_steps: number }) | (ExecutionEventBase & { type: 'STEP_STARTED'; step_index: number; step_name: string; capability: string; }) | (ExecutionEventBase & { type: 'STEP_COMPLETED'; step_index: number; outputs: Record<string, any>; duration_ms: number; }) | (ExecutionEventBase & { type: 'STEP_FAILED_TRANSIENT'; step_index: number; error: StepError; retry_in_ms: number; retry_count: number; }) | (ExecutionEventBase & { type: 'STEP_FAILED_PERMANENT'; step_index: number; error: StepError }) | (ExecutionEventBase & { type: 'STEP_RETRY'; step_index: number; retry_count: number }) | (ExecutionEventBase & { type: 'GATE_TRIGGERED'; step_index: number; gate: GateDescriptor; view_descriptor: ViewDescriptor; }) | (ExecutionEventBase & { type: 'GATE_RESOLVED'; step_index: number; resolution: GateResolution }) | (ExecutionEventBase & { type: 'INTERACTION_REQUIRED'; step_index: number; interaction: InteractionDescriptor; view_descriptor: ViewDescriptor; }) | (ExecutionEventBase & { type: 'INTERACTION_RECEIVED'; step_index: number }) | (ExecutionEventBase & { type: 'VIEW_UPDATE'; step_index: number; view_descriptor: ViewDescriptor; }) | (ExecutionEventBase & { type: 'OUTBOX_STAGED'; step_index: number; outbox_entry: OutboxEntry }) | (ExecutionEventBase & { type: 'OUTBOX_COMMITTING'; total_entries: number }) | (ExecutionEventBase & { type: 'OUTBOX_COMMITTED'; step_index: number; receipt: OutboxReceipt }) | (ExecutionEventBase & { type: 'EXECUTION_COMPLETED'; result: ExecutionResult }) | (ExecutionEventBase & { type: 'EXECUTION_FAILED'; error: ExecutionError });4. Checkpoint and Durability
Section titled “4. Checkpoint and Durability”4.1 Checkpoint Structure
Section titled “4.1 Checkpoint Structure”After every state transition, the engine persists the full execution state:
interface ExecutionCheckpoint { // Identity execution_id: string; plan_id: string; plan_version: string;
// State machine status: ExecutionStatus; current_step_index: number;
// Accumulated execution context (JSON-serializable) context_snapshot: { intent_inputs: Record<string, any>; step_outputs: Record<number, Record<string, any>>; interaction_responses: Record<number, Record<string, any>>; gate_resolutions: Record<number, GateResolution>; };
// Step results accumulated so far step_results: StepResult[];
// Retry state (populated when status is WAITING_RETRY) retry_info?: { step_index: number; retry_count: number; max_retries: number; next_retry_at: string; // ISO timestamp last_error: StepError; total_retry_wait_ms: number; // Cumulative time spent waiting across all retries };
// Staged outbox entries (committed only after all steps succeed) staged_outbox_entries: OutboxEntry[];
// Trace entries accumulated so far trace_entry_count: number; // Trace is written incrementally, not stored in checkpoint
// Configuration (captured at start, immutable) execution_mode: ExecutionMode; environment: string; user_context: UserContext; options: ExecutionOptions; plan: PlanIR; // The full plan (needed for recovery)
// Timing started_at: string; last_checkpoint_at: string; active_duration_ms: number; // Cumulative active execution time}4.2 Checkpoint Store
Section titled “4.2 Checkpoint Store”Checkpoints are persisted to a Zakardo-managed table in the developer’s database:
-- zakardo_executions table (managed by Zakardo runtime, created on init)CREATE TABLE zakardo_executions ( execution_id TEXT PRIMARY KEY, plan_id TEXT NOT NULL, status TEXT NOT NULL, -- current state machine state checkpoint JSONB NOT NULL, -- full ExecutionCheckpoint started_at TIMESTAMP NOT NULL, last_updated_at TIMESTAMP NOT NULL, completed_at TIMESTAMP,
-- Indexes for recovery and querying INDEX idx_status (status), INDEX idx_plan_id (plan_id), INDEX idx_started_at (started_at));The checkpoint store uses the developer’s existing database connection (the db service from service configuration). This means:
- No separate infrastructure for Zakardo state.
- Checkpoints live alongside the application’s data.
- The developer’s DB backup strategy covers Zakardo execution state.
For local development, if no database is configured, Zakardo falls back to SQLite (a file-based zakardo.db in the project directory).
4.3 Checkpoint Frequency
Section titled “4.3 Checkpoint Frequency”Checkpoints are written at every state transition, not after every internal operation:
| Event | Checkpoint Written? |
|---|---|
| Execution initialized | Yes (status: INITIALIZED) |
| Step completed | Yes (advance current_step_index, write step output) |
| Step failed (transient, will retry) | Yes (status: WAITING_RETRY, retry_info populated) |
| Step failed (permanent) | Yes (status: PERMANENTLY_FAILED) |
| Gate triggered | Yes (status: WAITING_GATE) |
| Gate resolved | Yes (advance to next phase of step) |
| Interaction required | Yes (status: WAITING_INTERACTION) |
| Interaction received | Yes (write response, advance) |
| All steps done | Yes (status: COMMITTING_OUTBOX) |
| Outbox committed | Yes (status: COMPLETED) |
| Mid-capability (during a long-running capability call) | No — capabilities are atomic from the engine’s perspective |
This means: if the process dies during a capability invocation, the engine will re-execute that step on recovery. Capabilities with side effects must be idempotent (see Section 11).
4.4 Process Recovery
Section titled “4.4 Process Recovery”On startup, the engine runs a recovery sweep:
async function recoverExecutions(): Promise<void> { // Find all non-terminal executions const active = await checkpointStore.findByStatus([ 'INITIALIZED', 'EXECUTING_STEP', // was mid-step when process died 'WAITING_RETRY', 'WAITING_GATE', 'WAITING_INTERACTION', 'COMMITTING_OUTBOX', ]);
for (const checkpoint of active) { switch (checkpoint.status) { case 'INITIALIZED': case 'EXECUTING_STEP': // Was mid-step when process died. Re-execute current step. // Capability must be idempotent (at-least-once semantics). scheduleAdvance(checkpoint.execution_id, 0); // immediate break;
case 'WAITING_RETRY': // Re-schedule the retry timer const delay = Math.max( 0, new Date(checkpoint.retry_info.next_retry_at).getTime() - Date.now(), ); scheduleAdvance(checkpoint.execution_id, delay); break;
case 'WAITING_GATE': case 'WAITING_INTERACTION': // Still waiting. Check if timeout has expired. if (hasTimedOut(checkpoint)) { await transitionToPermanentlyFailed(checkpoint, 'TIMEOUT'); } // Otherwise, do nothing. The response handler will wake us. break;
case 'COMMITTING_OUTBOX': // Was mid-outbox-commit. Retry (idempotency keys protect us). scheduleAdvance(checkpoint.execution_id, 0); break; } }}This recovery function runs once on startup. It ensures no execution is left in limbo.
4.5 Event Store
Section titled “4.5 Event Store”Every execution event is persisted to the zakardo_execution_events table before being emitted to connected clients. This ensures events are never lost — a client that disconnects and reconnects can replay missed events.
CREATE TABLE zakardo_execution_events ( event_id TEXT NOT NULL, -- Monotonic within execution (e.g. "evt_001") execution_id TEXT NOT NULL REFERENCES zakardo_executions(execution_id), event_type TEXT NOT NULL, -- e.g. "STEP_COMPLETED" event_data JSONB NOT NULL, -- Full event payload (JSON-serialized ExecutionEvent) created_at TIMESTAMP NOT NULL DEFAULT NOW(),
PRIMARY KEY (execution_id, event_id), INDEX idx_execution_events_type (execution_id, event_type));Event lifecycle:
State transition occurs │ ▼Build ExecutionEvent (assign next event_id) │ ▼INSERT INTO zakardo_execution_events ← Durable. Survives crashes. │ ▼Publish to EventChannel ← Best-effort. For connected clients.The Event Store is the source of truth for execution progress. The EventChannel is a convenience for live delivery.
Event replay on client connection:
When a client connects to the SSE stream (or polls via the events endpoint), the server:
- Reads all stored events for this execution from
zakardo_execution_events. - If the client provides a
Last-Event-ID, only events after that ID are replayed. - After replay, the server subscribes to the EventChannel for new live events.
Retention: Events are retained for the same duration as the parent execution’s checkpoint. When an execution is archived or purged, its events are purged with it.
4.6 Event Channel
Section titled “4.6 Event Channel”The Event Channel is the live delivery mechanism that pushes events from the executor (which may be a different process) to SSE handlers in the API process. It is implemented as an adapter, allowing different backends depending on the deployment model.
interface EventChannel { /** Publish an event. Called by the executor after storing to Event Store. */ publish(executionId: string, event: ExecutionEvent): Promise<void>;
/** Subscribe to events for an execution. Called by the SSE handler. */ subscribe(executionId: string, handler: (event: ExecutionEvent) => void): Subscription;}
interface Subscription { unsubscribe(): void;}Built-in implementations:
| Implementation | When Used | How It Works |
|---|---|---|
InProcessEventChannel | dispatch.mode: 'in-process' | Node.js EventEmitter. Zero-cost. Events delivered within the same process. |
RedisEventChannel | dispatch.mode: 'worker' | Redis Pub/Sub. Executor publishes; API process subscribes. Cross-process delivery. |
The EventChannel does not guarantee delivery (that’s the Event Store’s job). It is a best-effort push notification for connected clients.
4.7 Dispatch Model
Section titled “4.7 Dispatch Model”The Execution Engine can run execution work in-process (on the API server’s event loop) or in a separate worker process (via a job queue). This is controlled by configuration, not code changes. The state machine, checkpointing, capability invocation, and event emission code are identical in both modes.
4.7.1 Dispatcher Interface
Section titled “4.7.1 Dispatcher Interface”interface Dispatcher { /** * Dispatch an execution for processing. * In in-process mode: runs on the current event loop. * In worker mode: enqueues a job for a worker to pick up. */ dispatch(request: ExecutionRequest): Promise<void>;
/** * Schedule a delayed dispatch (used for retry timers and recovery). */ scheduleDispatch(executionId: string, delayMs: number): Promise<void>;
/** * Cancel a scheduled dispatch (used when cancelling executions). */ cancelScheduledDispatch(executionId: string): Promise<void>;}4.7.2 In-Process Mode
Section titled “4.7.2 In-Process Mode”class InProcessDispatcher implements Dispatcher { async dispatch(request: ExecutionRequest): Promise<void> { // Run on the event loop — does not block the HTTP response. // The HTTP handler has already returned 202 Accepted. setImmediate(() => advanceExecution(request.execution_id)); }
async scheduleDispatch(executionId: string, delayMs: number): Promise<void> { setTimeout(() => advanceExecution(executionId), delayMs); }
async cancelScheduledDispatch(executionId: string): Promise<void> { // Clear the scheduled timeout (tracked by execution_id) clearTimeout(this.scheduledTimers.get(executionId)); }}Characteristics:
- Zero extra infrastructure. No Redis, no queue, no worker process.
- Execution runs on the API process event loop via
setImmediate. - Retry timers use
setTimeout(lost on process restart — recovered by §4.4 recovery sweep). - Suitable for: development, small teams, low-scale production.
Limitation: All execution work shares CPU and memory with HTTP handling. Heavy plans may degrade API latency. Horizontal scaling pins each execution to one API instance.
4.7.3 Worker Mode
Section titled “4.7.3 Worker Mode”class WorkerDispatcher implements Dispatcher { constructor(private queue: JobQueue) {}
async dispatch(request: ExecutionRequest): Promise<void> { await this.queue.add('execution:advance', { executionId: request.execution_id, }); }
async scheduleDispatch(executionId: string, delayMs: number): Promise<void> { await this.queue.add( 'execution:advance', { executionId }, { delay: delayMs, }, ); }
async cancelScheduledDispatch(executionId: string): Promise<void> { await this.queue.remove('execution:advance', { executionId }); }}Worker process:
// worker.ts — runs as a separate Node.js processimport { createZakardoWorker } from '@zakardo/core';
const worker = createZakardoWorker({ config, capabilities: capabilities.all, concurrency: 5, // Process up to 5 executions in parallel});
worker.start();The worker process:
- Consumes jobs from the queue (BullMQ backed by Redis).
- Loads the execution checkpoint from the database.
- Calls
advanceExecution(executionId)— the exact same function the in-process mode uses. - Publishes events via the
RedisEventChannel. - Checkpoints after every transition (same as in-process mode).
Characteristics:
- Execution runs in a dedicated process — API process is never burdened.
- Workers scale independently: add more workers for more throughput.
- Retry timers use the queue’s
delayfeature — survives worker restarts. - Suitable for: production, multi-tenant, high-scale.
Requirement: Redis for BullMQ and Pub/Sub.
4.7.4 Configuration
Section titled “4.7.4 Configuration”import { defineExecutionConfig } from '@zakardo/core';
export const execution = defineExecutionConfig({ dispatch: { /* * 'in-process': Execution runs on the API event loop. * 'worker': Execution runs in a separate worker process via job queue. */ mode: env.ZAKARDO_DISPATCH_MODE ?? 'in-process',
// Worker mode configuration (ignored in in-process mode) worker: { queue: { provider: 'bullmq', connection: { host: env.REDIS_HOST ?? 'localhost', port: Number(env.REDIS_PORT ?? 6379), }, }, events: { provider: 'redis-pubsub', // Uses same Redis connection }, concurrency: Number(env.ZAKARDO_WORKER_CONCURRENCY ?? 5), }, },});4.7.5 Dispatch Mode Comparison
Section titled “4.7.5 Dispatch Mode Comparison”| Aspect | In-Process | Worker |
|---|---|---|
| Infrastructure | None | Redis |
| Execution location | API event loop | Separate process |
| Retry timers | setTimeout (ephemeral) | Queue delay (durable) |
| Event delivery | EventEmitter | Redis Pub/Sub |
| Horizontal scaling | Executions pinned to one instance | Workers pick from shared queue |
| Recovery on crash | §4.4 recovery sweep on restart | §4.4 sweep + queue re-delivery |
| Best for | Dev, prototyping, small apps | Production, multi-tenant, scale |
The Execution Engine code (advanceExecution, checkpoint writes, capability invocation, outbox management) is identical in both modes. Only the Dispatcher and EventChannel adapters change. This follows the same adapter pattern used for DataAdapter (Prisma/Drizzle) and AuthAdapter (Better Auth).
5. Execution Lifecycle
Section titled “5. Execution Lifecycle”5.1 Phase Overview
Section titled “5.1 Phase Overview”Phase 1: INITIALIZATION Create checkpoint, resolve services, begin Trace, emit EXECUTION_STARTED.
Phase 2: SEQUENTIAL STEP EXECUTION For step 0, 1, 2, ... N (in order, one at a time): 2a. Per-step policy evaluation 2b. Gate handling (if policy triggers a gate) → may pause 2c. Input resolution (bind inputs from context) 2d. Capability invocation 2e. Output capture + checkpoint 2f. Interaction handling (if capability has Interaction Spec) → may pause 2g. View composition (output views) 2h. Outbox staging (if external effect) 2i. Trace recording Any step may also → WAITING_RETRY (transient failure)
Phase 3: OUTBOX COMMIT Commit all staged external effects (with idempotency + retry).
Phase 4: FINALIZATION Compute plan outputs, finalize Trace, emit completion event.5.2 Phase 1: Initialization
Section titled “5.2 Phase 1: Initialization”async function startExecution(request: ExecutionRequest): Promise<string> { // 1. Generate execution_id const execution_id = generateId();
// 2. Resolve services for the current environment + mode const services = serviceRegistry.resolve(request.environment, request.execution_mode);
// 3. Create initial checkpoint const checkpoint: ExecutionCheckpoint = { execution_id, plan_id: request.plan.plan_id, plan_version: request.plan.version, status: 'INITIALIZED', current_step_index: 0, context_snapshot: { intent_inputs: request.intent_inputs, step_outputs: {}, interaction_responses: {}, gate_resolutions: {}, }, step_results: [], staged_outbox_entries: [], trace_entry_count: 0, execution_mode: request.execution_mode, environment: request.environment, user_context: request.user_context, options: request.options ?? defaults, plan: request.plan, started_at: new Date().toISOString(), last_checkpoint_at: new Date().toISOString(), active_duration_ms: 0, };
// 4. Persist checkpoint (execution now survives restarts) await checkpointStore.save(checkpoint);
// 5. Begin Trace await trace.record(execution_id, 'EXECUTION_STARTED', { plan_id: request.plan.plan_id, execution_mode: request.execution_mode, environment: request.environment, user_id: request.user_context.user_id, });
// 6. Emit event to frontend emit({ type: 'EXECUTION_STARTED', execution_id, plan_id: request.plan.plan_id, total_steps: request.plan.steps.length, });
// 7. Immediately begin executing first step await advanceExecution(execution_id);
return execution_id;}Service resolution selects the correct service implementations based on the multi-mode service configuration (see System Architecture, Section 6.3):
| Environment | Mode | db service resolves to |
|---|---|---|
| production | LIVE | Real PrismaClient with production DATABASE_URL |
| production | DRY_RUN | ReadOnlyWrapper around production PrismaClient |
| production | SIMULATED | InMemoryDB |
| development | LIVE | PrismaClient with local dev DB |
| development | TRAINING | InMemoryDB with fixtures |
5.3 Phase 2: Sequential Step Execution
Section titled “5.3 Phase 2: Sequential Step Execution”The core engine loop. Called by every trigger (new execution, step completed, retry timer, gate response, interaction response, process recovery).
async function advanceExecution(execution_id: string): Promise<void> { const checkpoint = await checkpointStore.load(execution_id); const step = checkpoint.plan.steps[checkpoint.current_step_index]; const services = serviceRegistry.resolve(checkpoint.environment, checkpoint.execution_mode);
// Build CapabilityContext for this step const capabilityContext: CapabilityContext = { execution_id: checkpoint.execution_id, execution_mode: checkpoint.execution_mode, environment: checkpoint.environment, user: checkpoint.user_context, services, trace: new TraceWriter(execution_id), error: createErrorFactory(execution_id), };
// --- 2a. PER-STEP POLICY EVALUATION --- const policyResult = await policyEngine.evaluateStep(step, checkpoint); await trace.record(execution_id, 'POLICY_EVALUATED', { step_index: step.step_index, result: policyResult, });
if (policyResult.action === 'DENY') { // Permanent failure -- policy blocks this step at runtime await transitionToPermanentlyFailed(checkpoint, { code: 'POLICY_DENIED', category: 'AUTHORIZATION_ERROR', message: policyResult.reason, }); return; }
// --- 2b. GATE HANDLING --- if (policyResult.action === 'GATE') { const gateDescriptor = buildGateDescriptor(step, policyResult); const viewDescriptor = await viewComposition.composeGateView(gateDescriptor);
emit({ type: 'GATE_TRIGGERED', execution_id, step_index: step.step_index, gate: gateDescriptor, view_descriptor: viewDescriptor, }); await trace.record(execution_id, 'GATE_TRIGGERED', { step_index: step.step_index, gate_type: gateDescriptor.gate_type, });
// Check v1 auto-DENY rule if (requiresRoleUserLacks(policyResult, checkpoint.user_context)) { await transitionToPermanentlyFailed(checkpoint, { code: 'GATE_AUTO_DENIED', category: 'AUTHORIZATION_ERROR', message: 'Gate requires approver role the current user does not have', }); return; }
// Pause and wait for gate response checkpoint.status = 'WAITING_GATE'; await checkpointStore.save(checkpoint); return; // Execution pauses here. Gate response handler will call advanceExecution. }
// --- 2c. INPUT RESOLUTION --- let resolvedInputs: Record<string, any>; try { resolvedInputs = resolveInputs(step, checkpoint.context_snapshot); } catch (bindingError) { await transitionToPermanentlyFailed(checkpoint, { code: 'BINDING_ERROR', category: 'SYSTEM_ERROR', message: bindingError.message, }); return; }
// --- 2d. CAPABILITY INVOCATION --- emit({ type: 'STEP_STARTED', execution_id, step_index: step.step_index, step_name: step.name, capability: step.capability, }); await trace.record(execution_id, 'STEP_STARTED', { step_index: step.step_index, inputs: resolvedInputs, });
const stepStartTime = Date.now();
try { const output = await withTimeout( invokeCapability(step, resolvedInputs, capabilityContext), checkpoint.options.step_timeout_ms ?? 30000, );
const stepDuration = Date.now() - stepStartTime;
// --- 2e. OUTPUT CAPTURE + CHECKPOINT --- checkpoint.context_snapshot.step_outputs[step.step_index] = output; checkpoint.step_results.push({ step_index: step.step_index, step_name: step.name, capability: step.capability, capability_version: step.capability_version, status: 'COMPLETED', started_at: new Date(stepStartTime).toISOString(), completed_at: new Date().toISOString(), duration_ms: stepDuration, outputs: output, retry_count: checkpoint.retry_info?.step_index === step.step_index ? checkpoint.retry_info.retry_count : 0, }); checkpoint.retry_info = undefined; // Clear retry state on success checkpoint.active_duration_ms += stepDuration;
emit({ type: 'STEP_COMPLETED', execution_id, step_index: step.step_index, outputs: output, duration_ms: stepDuration, }); await trace.record(execution_id, 'STEP_COMPLETED', { step_index: step.step_index, outputs: output, duration_ms: stepDuration, });
// --- 2f. INTERACTION HANDLING --- if (step.interaction_spec) { const interactionDescriptor = await viewComposition.composeInteractionView(step, output); emit({ type: 'INTERACTION_REQUIRED', execution_id, step_index: step.step_index, interaction: interactionDescriptor, view_descriptor: interactionDescriptor.view_descriptor, }); await trace.record(execution_id, 'INTERACTION_REQUIRED', { step_index: step.step_index });
checkpoint.status = 'WAITING_INTERACTION'; await checkpointStore.save(checkpoint); return; // Pause. Interaction response handler will call advanceExecution. }
// --- 2g. VIEW COMPOSITION --- const viewDescriptor = await viewComposition.composeOutputView(step, output); emit({ type: 'VIEW_UPDATE', execution_id, step_index: step.step_index, view_descriptor: viewDescriptor, });
// --- 2h. OUTBOX STAGING --- if (needsOutbox(step)) { const outboxEntry = createOutboxEntry(checkpoint, step, output); checkpoint.staged_outbox_entries.push(outboxEntry); emit({ type: 'OUTBOX_STAGED', execution_id, step_index: step.step_index, outbox_entry: outboxEntry, }); await trace.record(execution_id, 'OUTBOX_STAGED', { step_index: step.step_index, outbox_id: outboxEntry.outbox_id, }); }
// --- ADVANCE TO NEXT STEP OR FINALIZE --- if (checkpoint.current_step_index < checkpoint.plan.steps.length - 1) { checkpoint.current_step_index++; checkpoint.status = 'EXECUTING_STEP'; await checkpointStore.save(checkpoint); await advanceExecution(execution_id); // Continue immediately with next step } else { // All steps done -- proceed to outbox commit checkpoint.status = 'COMMITTING_OUTBOX'; await checkpointStore.save(checkpoint); await commitOutbox(checkpoint); } } catch (error) { const stepDuration = Date.now() - stepStartTime; const categorized = categorizeError(error);
// --- TRANSIENT FAILURE: WAIT AND RETRY --- if (isTransient(categorized) && retryBudgetRemaining(checkpoint, step)) { const retryCount = (checkpoint.retry_info?.retry_count ?? 0) + 1; const delay = computeBackoff(retryCount, step.error_policy);
checkpoint.retry_info = { step_index: step.step_index, retry_count: retryCount, max_retries: step.error_policy?.max_retries ?? defaultMaxRetries(categorized.category), next_retry_at: new Date(Date.now() + delay).toISOString(), last_error: categorized, total_retry_wait_ms: (checkpoint.retry_info?.total_retry_wait_ms ?? 0) + delay, }; checkpoint.status = 'WAITING_RETRY'; checkpoint.active_duration_ms += stepDuration; await checkpointStore.save(checkpoint);
emit({ type: 'STEP_FAILED_TRANSIENT', execution_id, step_index: step.step_index, error: categorized, retry_in_ms: delay, retry_count: retryCount, }); await trace.record(execution_id, 'STEP_FAILED_TRANSIENT', { step_index: step.step_index, error: categorized, retry_in_ms: delay, });
scheduleWakeup(execution_id, delay); // Timer will call advanceExecution return; }
// --- PERMANENT FAILURE: STOP --- checkpoint.active_duration_ms += stepDuration; await transitionToPermanentlyFailed(checkpoint, categorized); }}5.4 Phase 3: Outbox Commit
Section titled “5.4 Phase 3: Outbox Commit”After all steps succeed, the engine commits staged external effects:
async function commitOutbox(checkpoint: ExecutionCheckpoint): Promise<void> { const { execution_id, staged_outbox_entries } = checkpoint;
if (staged_outbox_entries.length === 0) { // No external effects to commit -- go straight to completion await finalizeExecution(checkpoint, 'COMPLETED'); return; }
emit({ type: 'OUTBOX_COMMITTING', execution_id, total_entries: staged_outbox_entries.length });
for (const entry of staged_outbox_entries) { if (entry.status === 'SUCCEEDED') continue; // Already committed (recovery case)
try { const receipt = await executeOutboxEntry(entry); entry.status = 'SUCCEEDED'; entry.receipt = receipt; entry.committed_at = new Date().toISOString();
emit({ type: 'OUTBOX_COMMITTED', execution_id, step_index: entry.step_index, receipt }); await trace.record(execution_id, 'OUTBOX_COMMITTED', { outbox_id: entry.outbox_id, receipt });
// Checkpoint after each outbox entry (recovery-safe) await checkpointStore.save(checkpoint); } catch (error) { // Outbox commit failure -- retry with backoff entry.retry_count++; if (entry.retry_count <= entry.max_retries) { entry.status = 'RETRYING'; await checkpointStore.save(checkpoint); const delay = computeBackoff(entry.retry_count); scheduleWakeup(execution_id, delay); return; // Will resume on wakeup } else { // Outbox entry exhausted retries -- this is a serious issue // Other entries may have already committed. Log and alert. entry.status = 'FAILED'; entry.receipt = { success: false, error: error.message, timestamp: new Date().toISOString(), }; await trace.record(execution_id, 'OUTBOX_FAILED', { outbox_id: entry.outbox_id, error: error.message, }); // Continue with remaining entries -- partial commit is better than full stop } } }
await finalizeExecution(checkpoint, 'COMPLETED');}Idempotency is critical here. Each outbox entry has an idempotency_key. If the process dies after committing entry 2 of 5, recovery will re-process entries 1-5, but entries 1-2 will be no-ops because the external system recognizes the idempotency key.
5.5 Phase 4: Finalization
Section titled “5.5 Phase 4: Finalization”async function finalizeExecution( checkpoint: ExecutionCheckpoint, status: 'COMPLETED' | 'PERMANENTLY_FAILED', error?: ExecutionError,): Promise<void> { // 1. Compute plan-level outputs (if completed) const outputs = status === 'COMPLETED' ? computePlanOutputs(checkpoint) : {};
// 2. Build final result const result: ExecutionResult = { execution_id: checkpoint.execution_id, plan_id: checkpoint.plan_id, plan_version: checkpoint.plan_version, status, execution_mode: checkpoint.execution_mode, outputs, trace_id: checkpoint.execution_id, // Trace ID = execution ID started_at: checkpoint.started_at, completed_at: new Date().toISOString(), duration_ms: Date.now() - new Date(checkpoint.started_at).getTime(), active_duration_ms: checkpoint.active_duration_ms, step_results: checkpoint.step_results, error, };
// 3. Finalize Trace await trace.record( checkpoint.execution_id, status === 'COMPLETED' ? 'EXECUTION_COMPLETED' : 'EXECUTION_FAILED', { result }, );
// 4. Update checkpoint to terminal state checkpoint.status = status; checkpoint.last_checkpoint_at = new Date().toISOString(); await checkpointStore.save(checkpoint);
// 5. Emit event if (status === 'COMPLETED') { emit({ type: 'EXECUTION_COMPLETED', execution_id: checkpoint.execution_id, result }); } else { emit({ type: 'EXECUTION_FAILED', execution_id: checkpoint.execution_id, error }); }
// 6. If plan was CANDIDATE: record execution result for lifecycle evaluation if (checkpoint.plan.metadata?.lifecycle_state === 'CANDIDATE') { await planLibrary.recordExecutionResult(checkpoint.plan_id, result); }}6. Execution Context
Section titled “6. Execution Context”The Execution Context is the accumulated data from step executions. In the durable model, it is part of the checkpoint (persisted as context_snapshot).
6.1 Structure
Section titled “6.1 Structure”interface ContextSnapshot { // Intent inputs provided by the user (immutable after initialization) intent_inputs: Record<string, any>;
// Step outputs, keyed by step_index step_outputs: Record<number, Record<string, any>>;
// Interaction responses, keyed by step_index interaction_responses: Record<number, Record<string, any>>;
// Gate resolutions, keyed by step_index gate_resolutions: Record<number, GateResolution>;}6.2 Binding Resolution
Section titled “6.2 Binding Resolution”When a step’s input binding needs to be resolved, the engine reads from the context snapshot:
| Binding Type | Pattern | Resolution |
|---|---|---|
| Prior step output | $.steps[N].out.field | context.step_outputs[N].field |
| Named reference | $.named.ref_name.field | Resolve via step’s output_name → step_outputs[index].field |
| Intent input | $.inputs.field | context.intent_inputs.field |
| Literal value | { "$literal": value } | value (used directly) |
| Transform | { "$transform": { fn, args } } | Apply transform function to resolved args |
| Interaction response | $.interactions[N].field | context.interaction_responses[N].field |
6.3 Serialization Requirement
Section titled “6.3 Serialization Requirement”Because the context snapshot is persisted in the checkpoint, all data must be JSON-serializable. This is guaranteed by design:
- Capability inputs and outputs conform to JSON Schema (the Type System).
- Intent inputs are JSON payloads.
- Interaction responses conform to JSON Schema (the Interaction Spec’s response_schema).
- Gate resolutions are simple objects.
No Date objects, no Map/Set, no circular references, no functions. All data is plain JSON.
7. Execution Modes in Detail
Section titled “7. Execution Modes in Detail”7.1 LIVE
Section titled “7.1 LIVE”Full execution with real side effects.
- All capabilities execute with their real (production/environment-appropriate) service implementations.
- Outbox entries are committed after all steps succeed (real external calls).
- Gates are enforced (user must respond).
- Interaction Points are enforced.
7.2 DRY_RUN
Section titled “7.2 DRY_RUN”Read-only execution for pre-flight validation.
- READ capabilities: execute normally (real data access).
- WRITE_INTERNAL capabilities: execute but services resolve to read-only wrappers. Writes are intercepted and logged but not committed.
- WRITE_EXTERNAL capabilities: capability executes (to produce output data), but the outbox entry is marked as
DRY_RUNand never committed. Recorded as “would have done.” - DESTRUCTIVE capabilities: blocked at policy level. If one somehow reaches execution, the engine treats it as a permanent failure.
DRY_RUN output includes a dry_run_report:
interface DryRunReport { would_write: Array<{ step_index: number; capability: string; effect_class: 'WRITE_INTERNAL' | 'WRITE_EXTERNAL'; description: string; data_preview?: Record<string, any>; }>; blocked: Array<{ step_index: number; capability: string; effect_class: 'DESTRUCTIVE'; reason: string; }>;}7.3 SIMULATED
Section titled “7.3 SIMULATED”No real side effects. Everything is mocked.
- Services resolve to mock/in-memory implementations (per multi-mode service config).
- All capabilities execute against mock services.
- Outbox entries are staged but never committed (discarded at finalization).
- Gates are auto-approved (no user interaction).
- Interaction Points use default/sample responses (or are auto-skipped).
- No checkpointing — SIMULATED runs execute in-memory without persistence (they’re fast, disposable, and don’t need recovery).
7.4 TRAINING
Section titled “7.4 TRAINING”A variant of SIMULATED designed for plan training and evaluation.
- Services resolve to training-specific implementations (fixture databases, known-good datasets).
- Execution is recorded with enhanced trace detail.
- Gates and Interaction Points are auto-resolved using training fixtures.
- Output is compared against expected outputs and scored.
- No checkpointing — same as SIMULATED, runs are in-memory.
8. Gate Handling
Section titled “8. Gate Handling”8.1 Gate Lifecycle
Section titled “8.1 Gate Lifecycle”Policy triggers GATE → Engine determines gate type → A2UI Composition builds Gate A2UI surface → Engine emits GATE_TRIGGERED event to frontend → Engine checkpoints with status WAITING_GATE → Plan pauses (entire plan is blocked -- sequential execution) → Frontend renders gate UI → User responds (approve, deny, provide input) → Frontend sends gate response to engine via HTTP/WebSocket → Engine loads checkpoint, validates response → Engine records resolution in Trace → If approved: checkpoint, resume execution at next phase of current step → If denied: transition to PERMANENTLY_FAILED8.2 Gate Types
Section titled “8.2 Gate Types”| Gate Type | What the User Sees | Response Expected | On Approval | On Denial |
|---|---|---|---|---|
APPROVAL | ”Step X requires approval. Proceed?” | Yes / No | Step executes | Plan permanently failed |
INPUT | ”Step X needs additional information” | Form data | Data written to context; step executes | Plan permanently failed |
REVIEW | ”Please review before proceeding” | Acknowledged | Step executes | Plan permanently failed |
LOG_ONLY | Nothing (not surfaced to user) | None | Step executes immediately | N/A |
8.3 v1 Gate Model
Section titled “8.3 v1 Gate Model”In v1, gates are synchronous and single-user:
- The gate is presented to the current user (the one who initiated the plan).
- If the gate requires approver roles that the current user does not have: auto-DENY → plan permanently failed.
- The user must respond within
gate_timeout_ms. If they don’t: auto-DENY → plan permanently failed. - Gate state survives process restarts. The checkpoint is persisted with status
WAITING_GATE. When the user eventually responds (after a restart), the engine loads the checkpoint and resumes.
The “Plans always succeed” principle means the Planner should have detected that the user can’t approve the gate and offered an alternative (Request Pattern). Gates at execution time are for cases where the Planner determined the user CAN approve — they just need to explicitly do so.
8.4 v2 Gate Model (Future)
Section titled “8.4 v2 Gate Model (Future)”v2 will introduce:
- Async gates: gate response can come hours or days later.
- Cross-role gates: gate is routed to a user with the required role, not just the current user.
- Gate queues: pending gates visible in a dashboard for approvers.
- Gate delegation: current user can delegate the gate to a specific approver.
The durable state machine already supports this — the checkpoint persists the WAITING_GATE state indefinitely. v2 simply adds the routing and notification layer.
8.5 Gate A2UI Surface
Section titled “8.5 Gate A2UI Surface”interface GateDescriptor { gate_type: 'APPROVAL' | 'INPUT' | 'REVIEW' | 'LOG_ONLY'; step_index: number; step_name: string; capability: string; effect_class: string; policy_id: string; policy_description: string; context_summary: string; input_schema?: JSONSchema; // For INPUT gates review_data?: Record<string, any>; // For REVIEW gates}9. Interaction Point Handling
Section titled “9. Interaction Point Handling”9.1 Interaction Lifecycle
Section titled “9.1 Interaction Lifecycle”Capability executes and produces output → Engine checks if capability has Interaction Spec → If yes: A2UI Composition builds Interaction A2UI surface → Engine emits INTERACTION_REQUIRED event to frontend → Engine checkpoints with status WAITING_INTERACTION → Plan pauses (entire plan is blocked -- sequential execution) → Frontend renders interaction UI → User responds (select, input, confirm, etc.) → Frontend sends response to engine → Engine loads checkpoint, validates response against Interaction Spec response_schema → If valid: write response to context, checkpoint, resume → If invalid: request retry from frontend9.2 Interaction vs. Gate
Section titled “9.2 Interaction vs. Gate”| Gate | Interaction Point | |
|---|---|---|
| Triggered by | Policy | Capability’s Interaction Spec |
| Purpose | Governance (approve, review) | Functional (select, input, compare) |
| Data flow | Gate resolution (approved/denied) | User’s response data flows into next steps |
| Can block plan? | Yes (if denied) | Yes (if timeout or invalid response) |
| Skip in SIMULATED? | Auto-approve | Auto-resolve with defaults/fixtures |
| Survives restart? | Yes (checkpointed) | Yes (checkpointed) |
9.3 Interaction A2UI Surface
Section titled “9.3 Interaction A2UI Surface”interface InteractionDescriptor { interaction_type: 'SELECT' | 'COMPARE' | 'REVIEW' | 'INPUT' | 'CONFIRM'; step_index: number; step_name: string; capability: string; data: Record<string, any>; response_schema: JSONSchema; view_descriptor: ViewDescriptor;}10. Outbox
Section titled “10. Outbox”The Outbox manages external side effects for capabilities with REMOTE boundary and WRITE_EXTERNAL or DESTRUCTIVE effect class.
10.1 Deferred Commit Model
Section titled “10.1 Deferred Commit Model”In v1, external effects are staged during execution but only committed after all steps succeed. This is a critical design decision:
Step 1: DB write (committed by capability -- internal)Step 2: Stripe charge (staged in Outbox -- NOT called yet)Step 3: DB write (committed by capability -- internal)Step 4: FAILS (permanent)
→ Outbox.discardAll() → Stripe was NEVER called→ DB writes from steps 1 and 3 ARE committed (capability's responsibility)If all steps succeed:
Step 1-5: all succeed→ COMMITTING_OUTBOX phase→ Stripe is called, webhook is sent, etc.→ Each outbox entry committed with idempotency key→ If commit fails: retry with backoff→ After all committed: COMPLETEDImplication: No step can depend on the result of an external call made by a prior step within the same plan. If step 3 calls Stripe and step 5 needs the Stripe charge ID, this doesn’t work — the Stripe call hasn’t happened yet. In such cases, the capability should be designed to produce the charge ID during the commit phase, or the plan should be split into two plans connected by an async workflow (v2).
10.2 Why Deferred Commit?
Section titled “10.2 Why Deferred Commit?”- Partial execution safety. If any step fails, no external effects have been realized. You don’t need to compensate Stripe for a charge that never happened.
- Simpler failure model. The only state that needs manual recovery is internal DB writes from earlier steps. External systems are clean.
- Natural fit for sequential execution. Since steps run one at a time and external calls happen at the end, there’s a clean separation between “figure out what to do” and “do it.”
10.3 Outbox in Different Modes
Section titled “10.3 Outbox in Different Modes”| Mode | Outbox Behavior |
|---|---|
| LIVE | Stage during execution → Commit after all steps succeed → Execute external calls → Record receipts |
| DRY_RUN | Stage during execution → Record as “would commit” → Never execute |
| SIMULATED | Stage during execution → Discard at finalization |
| TRAINING | Stage during execution → Discard or execute against sandbox (per training config) |
10.4 Outbox Entry
Section titled “10.4 Outbox Entry”interface OutboxEntry { outbox_id: string; execution_id: string; step_index: number; capability: string; idempotency_key: string; // execution_id + step_index + capability (default) action: string; // Human-readable description payload: Record<string, any>; // Data for the external call status: 'STAGED' | 'COMMITTING' | 'SUCCEEDED' | 'FAILED' | 'RETRYING' | 'DISCARDED'; created_at: string; committed_at?: string; receipt?: OutboxReceipt; retry_count: number; max_retries: number; // Default: 5}
interface OutboxReceipt { success: boolean; external_id?: string; response_data?: Record<string, any>; error?: string; timestamp: string;}11. Idempotency
Section titled “11. Idempotency”11.1 The At-Least-Once Problem
Section titled “11.1 The At-Least-Once Problem”Because the engine checkpoints at state transitions (not mid-capability), a process crash during capability execution means the step will be re-executed on recovery. This is at-least-once semantics.
For READ capabilities, this is harmless.
For WRITE_INTERNAL and WRITE_EXTERNAL capabilities, re-execution could duplicate side effects unless the capability is idempotent.
11.2 Idempotency Key
Section titled “11.2 Idempotency Key”The engine provides a natural idempotency key to every capability via the CapabilityContext:
// Available in contextconst idempotencyKey = `${context.execution_id}:${step.step_index}`;Capability authors should use this key when performing side effects:
// Example: idempotent database writeexport default defineCapability(async (input, context) => { const db = context.services.db; const idempotencyKey = `${context.execution_id}:create_order`;
// Upsert with idempotency key -- safe to call twice const order = await db.order.upsert({ where: { idempotency_key: idempotencyKey }, create: { ...input, idempotency_key: idempotencyKey }, update: {}, // No-op if already exists });
return { order_id: order.id };});11.3 Outbox Idempotency
Section titled “11.3 Outbox Idempotency”Outbox entries have built-in idempotency keys. The Outbox commit phase passes the idempotency key to the external service. If the service supports idempotency (Stripe, most modern APIs), duplicate commits are harmless.
11.4 Authoring Guideline
Section titled “11.4 Authoring Guideline”Capabilities with side effects (WRITE_INTERNAL, WRITE_EXTERNAL, DESTRUCTIVE) should be designed for at-least-once execution. This is a capability authoring best practice, not an engine enforcement. The engine provides the idempotency key; the capability author uses it.
12. Error Handling
Section titled “12. Error Handling”12.1 Error Categories
Section titled “12.1 Error Categories”Capability errors are categorized to drive automatic behavior:
| Category | Nature | Meaning | Engine Behavior |
|---|---|---|---|
TRANSIENT | Temporary | Network timeout, connection refused, rate limited, service 503 | Retry with backoff |
EXTERNAL_ERROR | Temporary | External service returned a retryable error (5xx, timeout) | Retry with backoff |
TIMEOUT | Temporary | Step exceeded its timeout | Retry once, then permanent fail |
SYSTEM_ERROR | Ambiguous | Internal error, unhandled exception | Retry once, then permanent fail |
BUSINESS_ERROR | Permanent | Business rule prevents completion (insufficient funds, account locked) | Permanent fail immediately |
VALIDATION_ERROR | Permanent | Input data invalid at runtime | Permanent fail immediately |
AUTHORIZATION_ERROR | Permanent | Runtime permission check failed | Permanent fail immediately |
12.2 Transient vs. Permanent
Section titled “12.2 Transient vs. Permanent”The engine’s core distinction:
- Transient: The step failed because of a temporary condition. If we wait and try again, it will likely succeed. The plan enters
WAITING_RETRY. - Permanent: The step failed because of a condition that won’t change by retrying. The plan enters
PERMANENTLY_FAILED.
12.3 Retry Configuration
Section titled “12.3 Retry Configuration”Each step can declare retry configuration. If not declared, defaults apply:
interface StepRetryConfig { max_retries: number; // How many times to retry initial_delay_ms: number; // First retry delay backoff: 'linear' | 'exponential'; // Backoff strategy max_delay_ms: number; // Cap on backoff delay}Defaults by error category:
| Category | max_retries | initial_delay_ms | backoff | max_delay_ms |
|---|---|---|---|---|
| TRANSIENT | 3 | 1,000 | exponential | 30,000 |
| EXTERNAL_ERROR | 3 | 2,000 | exponential | 60,000 |
| TIMEOUT | 1 | 5,000 | linear | 5,000 |
| SYSTEM_ERROR | 1 | 2,000 | linear | 2,000 |
| BUSINESS_ERROR | 0 | N/A | N/A | N/A |
| VALIDATION_ERROR | 0 | N/A | N/A | N/A |
| AUTHORIZATION_ERROR | 0 | N/A | N/A | N/A |
12.4 Retry Budget
Section titled “12.4 Retry Budget”The retry_budget_ms option (default: 300,000ms = 5 min) limits the total time spent retrying across all steps in a single execution. This prevents a plan from retrying indefinitely if a service is down for a long time.
When the retry budget is exhausted, the next transient failure becomes a permanent failure.
12.5 Error on Permanent Failure
Section titled “12.5 Error on Permanent Failure”When a step permanently fails:
- All staged outbox entries are discarded (external effects never committed).
- The Trace records the failure (step, error, retry history).
- The checkpoint transitions to
PERMANENTLY_FAILED. - The
EXECUTION_FAILEDevent is emitted to the frontend. - Internal DB writes from prior steps remain committed (this is the developer’s responsibility — see Section 11 on idempotency and Section 13 on compensation).
13. Compensation Strategy
Section titled “13. Compensation Strategy”13.1 v1 Model: Deferred Outbox + Observability
Section titled “13.1 v1 Model: Deferred Outbox + Observability”v1 does not provide automatic compensation (saga/rollback). Instead, it minimizes the need for compensation through architectural design:
| Effect Type | v1 Compensation Strategy |
|---|---|
| READ | No side effects. Nothing to compensate. |
| WRITE_INTERNAL | Capability author’s responsibility. Capabilities manage their own DB transactions. If a later step fails, earlier DB writes remain. The Trace records exactly what was written. |
| WRITE_EXTERNAL | Deferred commit via Outbox. External calls only happen after all steps succeed. If any step fails, the outbox is discarded — no external compensation needed. |
| DESTRUCTIVE | Policy requires gates (user explicitly approves). DRY_RUN first (policy can enforce). The user knows what they’re approving. |
Why this works for v1:
- External effects (the hardest to compensate) are never committed until the plan fully succeeds.
- Internal DB writes are the developer’s domain — they control the ORM, transactions, and data model.
- The Trace provides complete data for any manual recovery needed.
- The validation pipeline (Planner + Validator) catches most issues before execution. Runtime failures are primarily transient (handled by retry).
13.2 What the Developer Can Do
Section titled “13.2 What the Developer Can Do”For plans where internal state consistency is critical, developers can:
- Use database transactions within capabilities. A capability can start a transaction and commit/rollback within its own scope.
- Design capabilities to be safe. Prefer soft deletes over hard deletes. Use status fields (“pending”, “confirmed”) instead of immediate state changes.
- Build recovery plans. If plan A partially fails, the developer can author a “recover from plan A failure” plan that reverses the internal changes.
- Use DRY_RUN first. Run the plan in DRY_RUN mode to validate the full flow before running in LIVE mode. Policy can enforce this for certain plan types.
13.3 v2 Compensation (Future)
Section titled “13.3 v2 Compensation (Future)”v2 may introduce:
- Compensation functions: each capability optionally declares a
compensatehandler. - Saga orchestration: the engine runs compensation handlers in reverse order for completed steps when a later step permanently fails.
- Compensation tracing: every compensation attempt is recorded.
- Partial compensation: handle the case where compensation itself fails (dead letter queue, manual intervention).
The durable state machine architecture already supports this — compensation is just another phase in the state machine (between PERMANENTLY_FAILED and a new COMPENSATED or COMPENSATION_FAILED terminal state).
14. USER_ENV Capabilities
Section titled “14. USER_ENV Capabilities”Capabilities with USER_ENV boundary execute in the user’s browser, not on the server.
14.1 Execution Flow
Section titled “14.1 Execution Flow”Engine reaches a USER_ENV step → Serialize the step's resolved inputs → Emit USER_ENV_EXECUTE event to frontend via WebSocket/SSE → Engine checkpoints with status WAITING_INTERACTION (reuses interaction wait) → Plan pauses → Frontend SDK receives the event → Frontend SDK loads and executes the capability's client-side implementation → Capability executes in the browser (DOM manipulation, local storage, etc.) → Frontend SDK sends output back to engine → Engine loads checkpoint, writes output to context → Execution resumes14.2 Constraints
Section titled “14.2 Constraints”- USER_ENV capabilities don’t call external APIs directly (if they need external data, it comes from prior step outputs).
step_timeout_msapplies (accounting for network round-trip).- In SIMULATED and TRAINING modes, USER_ENV capabilities are mocked on the server side (no browser interaction).
15. Isolation
Section titled “15. Isolation”15.1 Plan-Level Isolation
Section titled “15.1 Plan-Level Isolation”Each plan execution is fully isolated:
- Separate
execution_idandExecutionCheckpoint. - Separate Trace.
- Separate staged Outbox.
- Shared services (but services are designed for concurrent access — they’re DB clients, API clients, etc.).
Multiple plans can execute concurrently for different users or the same user. Each has its own checkpoint. There is no global execution lock.
15.2 No Step-Level Concurrency (v1)
Section titled “15.2 No Step-Level Concurrency (v1)”v1 is sequential. Only one step executes at a time. No concurrent writes to the context. No race conditions between steps.
v2 may introduce parallel execution for steps without data dependencies. The checkpoint model would extend to track multiple in-progress steps.
16. Timeouts
Section titled “16. Timeouts”| Timeout | Default | Applies To | On Expiry |
|---|---|---|---|
| Overall execution | 300,000ms (5 min) | Active step execution time (excludes wait time) | PERMANENTLY_FAILED |
| Step execution | 30,000ms (30s) | Single capability invocation | Categorized as TIMEOUT, retry once then fail |
| Gate response | 3,600,000ms (1 hour) | Time in WAITING_GATE state | Auto-DENY → PERMANENTLY_FAILED |
| Interaction response | 3,600,000ms (1 hour) | Time in WAITING_INTERACTION state | PERMANENTLY_FAILED |
| Retry budget | 300,000ms (5 min) | Cumulative time in WAITING_RETRY across all steps | Next transient → permanent |
All timeouts are configurable per execution via ExecutionOptions and per plan via Plan IR metadata.
Important: The overall execution timeout counts active execution time only (time spent in capability invocations). Time spent in WAITING_GATE, WAITING_INTERACTION, and WAITING_RETRY does not count against the overall timeout. This prevents governance and infrastructure overhead from killing valid executions. The retry_budget_ms separately caps retry wait time.
17. Execution Hooks
Section titled “17. Execution Hooks”Developers can register lifecycle hooks:
{ execution: { hooks: { beforeExecution?: (request: ExecutionRequest) => Promise<void>; afterExecution?: (result: ExecutionResult) => Promise<void>; beforeStep?: (step: PlanStep, checkpoint: ExecutionCheckpoint) => Promise<void>; afterStep?: (step: PlanStep, result: StepResult, checkpoint: ExecutionCheckpoint) => Promise<void>; onGateTriggered?: (gate: GateDescriptor, checkpoint: ExecutionCheckpoint) => Promise<void>; onGateResolved?: (gate: GateDescriptor, resolution: GateResolution) => Promise<void>; onRetry?: (step: PlanStep, error: StepError, retryCount: number) => Promise<void>; onPermanentFailure?: (step: PlanStep, error: StepError, checkpoint: ExecutionCheckpoint) => Promise<void>; } }}Hooks are for observability, logging, and external integrations. They cannot modify the execution flow (they don’t return values the engine uses for decisions). If a hook throws, the error is logged in the Trace but does not affect execution.
18. Execution Trace Integration
Section titled “18. Execution Trace Integration”Every state transition writes to the Trace:
| Event | Recorded Data |
|---|---|
| Execution started | plan_id, execution_mode, environment, user_id, timestamp |
| Step started | step_index, capability, resolved inputs, timestamp |
| Policy evaluated | step_index, policy results (allow/deny/gate/warn) |
| Gate triggered | step_index, gate type, policy_id |
| Gate resolved | step_index, resolution (approved/denied), duration |
| Capability completed | step_index, outputs, duration_ms |
| Capability failed (transient) | step_index, error, retry_in_ms, retry_count |
| Capability failed (permanent) | step_index, error, total_retries |
| Retry attempted | step_index, retry_count, previous_error |
| Outbox staged | step_index, outbox_id, action |
| Outbox committed | outbox_id, receipt |
| Outbox failed | outbox_id, error, retry_count |
| Interaction required | step_index, interaction type |
| Interaction received | step_index, response data |
| View composed | step_index, view_descriptor type (anchored/ephemeral) |
| Checkpoint saved | status, current_step_index, active_duration_ms |
| Execution completed | final status, total duration, active duration, output summary |
19. Summary of Decisions
Section titled “19. Summary of Decisions”| Decision | Resolution |
|---|---|
| Sequential execution in v1 | Steps execute one at a time, in order. No parallel execution. Pauses block the entire plan. Parallel execution is a future enhancement. |
| Durable state machine | Execution state is checkpointed to persistent storage after every state transition. Executions survive process restarts. |
| Checkpoint store | Zakardo-managed table (zakardo_executions) in the developer’s database. SQLite fallback for local dev. |
| Process recovery | On startup, the engine recovers all non-terminal executions: re-executes interrupted steps, re-schedules retry timers, checks timeouts on waiting states. |
| Transient vs. permanent failure | Core distinction. Transient failures (network, rate limit, 503) → automatic retry with backoff. Permanent failures (business rule, auth, validation) → immediate stop. |
| Retry with backoff | Transient failures trigger WAITING_RETRY state with configurable backoff (default: exponential, 3 retries, 30s cap). Retry budget caps total retry time. |
| Deferred outbox commit | External effects are staged during execution but only committed after ALL steps succeed. If any step permanently fails, outbox is discarded. |
| Idempotency requirement | Capabilities with side effects should be idempotent. Engine provides execution_id:step_index as natural idempotency key. |
| No automatic compensation in v1 | Internal DB writes from completed steps are not rolled back on later failure. External effects are protected by deferred commit. Developer handles internal recovery via Trace data. |
| Event-driven output | Engine produces events via store-then-emit pattern. Events stored to zakardo_execution_events first, then pushed via EventChannel. Clients consume via SSE or polling endpoint. |
| Per-step policy evaluation | Every step is policy-checked at execution time. Defense-in-depth behind the Planner’s pre-generation checks. |
| v1 gates are synchronous | Gates are presented to the current user. Auto-DENY if user lacks approver roles. Gate state survives restarts (checkpointed). |
| Mode changes everything | Service resolution, outbox behavior, gate handling, interaction handling, and checkpointing all depend on execution mode. SIMULATED and TRAINING skip checkpointing. |
| Gate/interaction timeout separate from execution timeout | Wait time in gates, interactions, and retries does not count against overall execution timeout. |
| DRY_RUN report | DRY_RUN produces a report of “would have done” writes. Reads execute normally. |
| Trace everything | Every state transition, step result, policy check, gate, interaction, retry, outbox entry, and timing is recorded. |
| Dispatch model (hybrid) | Two modes: in-process (zero infrastructure, API event loop) and worker (BullMQ + Redis, separate process). Same state machine code in both. Adapter pattern. |
| Event Store | All events persisted to zakardo_execution_events before emitting. Enables client reconnection, catch-up replay, and polling. Events purged with parent execution. |
| Event Channel adapter | InProcessEventChannel (Node.js EventEmitter) for in-process mode. RedisEventChannel (Redis Pub/Sub) for worker mode. Best-effort delivery; Event Store is source of truth. |
20. Questions Deferred
Section titled “20. Questions Deferred”| Question | Deferred To |
|---|---|
| What is the exact Trace entry format? | Trace System Specification |
| How does the frontend render A2UI messages and handle events? | Frontend Architecture Specification |
| How does the Plan Library record execution results for lifecycle evaluation? | Plan Library and Lifecycle Specification |
| How are execution hooks registered and managed? | Runtime API and SDK Specification |
| How do training runs compare outputs to expected results? | Training and Validation Specification |
| How does the CLI support local execution for development? | CLI and DX Specification |
| How are execution metrics monitored and alerted on? | Trace System Specification |
| What is the compensation/saga model for v2? | Future planning (v2 scope) |
| How does the checkpoint store migration work? | Runtime API and SDK Specification |