Skip to content

Execution Engine

The Execution Engine is the component that runs validated Plans. It transforms a static Plan IR document into a live, governed, observable process — invoking capabilities sequentially, managing data flow, enforcing policies per-step, handling gates and interaction points, recording traces, and managing the outbox for external effects. The engine is a durable state machine with persistent checkpointing: plan executions survive process restarts, transient failures are retried automatically, and external effects are deferred until all steps succeed. This specification defines its architecture, execution lifecycle, durability model, error handling, and integration with every other Zakardo subsystem.


  1. Execute what the plan says. The Execution Engine does not reason about goals or choose alternatives. It runs the validated Plan IR faithfully. All reasoning happened in the Planner; all correctness was verified by the Validator.
  2. Sequential execution in v1. Steps execute one at a time, in order. Pauses (gates, interactions, retries) block the entire plan. This is intentionally simple. Parallel execution is a future enhancement.
  3. Durable by default. Every state transition is checkpointed to persistent storage. Process restarts do not lose executions. Paused plans survive indefinitely. Retrying plans pick up where they left off.
  4. Transient failures are invisible to users. Network blip? Service momentarily down? The engine waits and retries. The user sees “retrying step 3…” not “plan failed.”
  5. Every step is governed. Per-step policy evaluation means that even if a plan passed validation, runtime conditions (rate limits, dynamic constraints) can still gate or block individual steps.
  6. Every step is observed. Every capability invocation, gate resolution, interaction response, error, retry, and timing is recorded in the Trace. No silent execution.
  7. Execution modes are real. SIMULATED, DRY_RUN, LIVE, and TRAINING are not labels — they fundamentally change what the engine does. Services are mode-resolved, outbox behavior changes, and capabilities may receive entirely different implementations.
  8. External effects are deferred. The Outbox stages external side effects during execution but only commits them after all steps succeed. If any step permanently fails, no external effects are realized.
  9. Errors are structured and categorized. Transient vs. permanent failure distinction drives automatic behavior: retry the transient, stop on the permanent.

The execution engine manages each plan execution as a persistent state machine. Instead of a single async function that runs to completion (and dies if the process restarts), the engine:

  1. Advances the state machine one transition at a time.
  2. Checkpoints the full execution state to persistent storage after every transition.
  3. Can be woken up by triggers: timers (retry), external signals (gate response, interaction response), or process recovery.

This means:

  • A plan paused at a gate survives server restarts.
  • A plan waiting to retry a transient failure resumes automatically.
  • No execution data is lost unless the checkpoint store itself is lost.
INITIALIZED
│
▼
EXECUTING_STEP ◄──────────────────┐
│ │
├── step succeeds ─────────────┤ (checkpoint, advance to next step)
│ │
├── transient failure ───► WAITING_RETRY
│ │ (backoff timer)
│ │ timer expires → re-execute step
│ │
├── gate triggered ────────► WAITING_GATE
│ │ (user responds → resume)
│ │
├── interaction required ──► WAITING_INTERACTION
│ │ (user responds → resume)
│ │
└── permanent failure ─────► PERMANENTLY_FAILED
│
▼
(discard staged outbox, finalize trace)
All steps completed ──────────► COMMITTING_OUTBOX
│
├── outbox commit succeeds ──► COMPLETED
│
└── outbox commit fails ──► COMMITTING_OUTBOX
(retry with idempotency keys)

Terminal states: COMPLETED, PERMANENTLY_FAILED

Waiting states: WAITING_RETRY, WAITING_GATE, WAITING_INTERACTION — the plan is alive but paused. The checkpoint is persisted. A trigger will resume it.

TriggerSourceWakes State
New execution submittedHTTP request from Runtime APICreates INITIALIZED, immediately advances to EXECUTING_STEP
Step completedInternal (engine loop)EXECUTING_STEP → EXECUTING_STEP (next step) or COMMITTING_OUTBOX
Retry timer expiresScheduled timer (setTimeout in-process, or external scheduler)WAITING_RETRY → EXECUTING_STEP
Gate response receivedHTTP/WebSocket from frontendWAITING_GATE → EXECUTING_STEP
Interaction response receivedHTTP/WebSocket from frontendWAITING_INTERACTION → EXECUTING_STEP
Process recoveryStartup recovery sweepAny non-terminal state → re-evaluate and advance
Outbox commit completedInternalCOMMITTING_OUTBOX → COMPLETED

interface ExecutionRequest {
plan: PlanIR; // Validated Full Plan IR
validation_id: string; // Proof of validation
user_context: UserContext; // Who is executing
execution_mode: ExecutionMode; // LIVE, DRY_RUN, SIMULATED, TRAINING
environment: string; // 'production', 'staging', 'development'
intent_inputs: Record<string, any>; // User-provided inputs that satisfy plan.input_schema
options?: ExecutionOptions;
}
interface ExecutionOptions {
timeout_ms?: number; // Overall execution timeout (default: 300000 = 5 min)
step_timeout_ms?: number; // Per-step timeout (default: 30000 = 30s)
gate_timeout_ms?: number; // How long to wait for gate responses (default: 3600000 = 1 hour)
interaction_timeout_ms?: number; // How long to wait for interaction responses (default: 3600000 = 1 hour)
retry_budget_ms?: number; // Max total time spent in WAITING_RETRY across all steps (default: 300000 = 5 min)
trace_level?: 'minimal' | 'standard' | 'verbose'; // Trace detail level
}
type ExecutionMode = 'LIVE' | 'DRY_RUN' | 'SIMULATED' | 'TRAINING';

The Execution Engine emits a stream of events and produces a final result when execution reaches a terminal state.

// Final result (available after execution reaches terminal state)
interface ExecutionResult {
execution_id: string;
plan_id: string;
plan_version: string;
status: ExecutionStatus;
execution_mode: ExecutionMode;
outputs: Record<string, any>; // Plan-level outputs (from output_schema mapping)
trace_id: string; // Reference to the full Trace
started_at: string;
completed_at: string;
duration_ms: number; // Wall clock time from start to finish
active_duration_ms: number; // Time spent actually executing (excludes waits)
step_results: StepResult[];
dry_run_report?: DryRunReport; // Only for DRY_RUN mode
error?: ExecutionError; // If status is PERMANENTLY_FAILED
}
type ExecutionStatus =
| 'COMPLETED' // All steps succeeded, outbox committed
| 'PERMANENTLY_FAILED' // A step permanently failed
| 'WAITING_RETRY' // Paused, waiting to retry a transient failure
| 'WAITING_GATE' // Paused at a governance gate
| 'WAITING_INTERACTION' // Paused at an interaction point
| 'COMMITTING_OUTBOX'; // All steps done, committing external effects
interface StepResult {
step_index: number;
step_name: string;
capability: string;
capability_version: string;
status: StepStatus;
started_at: string;
completed_at?: string;
duration_ms?: number;
outputs?: Record<string, any>;
error?: StepError;
gate_resolution?: GateResolution;
interaction_response?: InteractionResponse;
policy_evaluation?: PolicyEvalResult;
retry_count: number;
}
type StepStatus =
| 'PENDING' // Not yet started
| 'COMPLETED' // Finished successfully
| 'FAILED' // Permanently failed (retries exhausted or non-retryable)
| 'RETRYING' // Failed transiently, will retry
| 'GATED' // Paused at a gate
| 'INTERACTING' // Paused at an interaction point
| 'ABORTED'; // User denied gate or aborted interaction

During execution, the engine produces events for every significant state transition. Events follow a store-then-emit pattern: every event is first persisted to the Event Store (see §4.5), then emitted to the Event Channel (see §4.6) for any connected clients. This guarantees that:

  • Events are never lost, even if no client is connected.
  • Clients can reconnect and catch up by replaying stored events.
  • The Event Store is the source of truth; the live channel is a convenience.

Every event carries a monotonically increasing event_id within its execution scope:

interface ExecutionEventBase {
event_id: string; // Monotonic ID within this execution (e.g. "evt_001")
execution_id: string;
timestamp: string; // ISO 8601
}
type ExecutionEvent =
| (ExecutionEventBase & { type: 'EXECUTION_STARTED'; plan_id: string; total_steps: number })
| (ExecutionEventBase & {
type: 'STEP_STARTED';
step_index: number;
step_name: string;
capability: string;
})
| (ExecutionEventBase & {
type: 'STEP_COMPLETED';
step_index: number;
outputs: Record<string, any>;
duration_ms: number;
})
| (ExecutionEventBase & {
type: 'STEP_FAILED_TRANSIENT';
step_index: number;
error: StepError;
retry_in_ms: number;
retry_count: number;
})
| (ExecutionEventBase & { type: 'STEP_FAILED_PERMANENT'; step_index: number; error: StepError })
| (ExecutionEventBase & { type: 'STEP_RETRY'; step_index: number; retry_count: number })
| (ExecutionEventBase & {
type: 'GATE_TRIGGERED';
step_index: number;
gate: GateDescriptor;
view_descriptor: ViewDescriptor;
})
| (ExecutionEventBase & { type: 'GATE_RESOLVED'; step_index: number; resolution: GateResolution })
| (ExecutionEventBase & {
type: 'INTERACTION_REQUIRED';
step_index: number;
interaction: InteractionDescriptor;
view_descriptor: ViewDescriptor;
})
| (ExecutionEventBase & { type: 'INTERACTION_RECEIVED'; step_index: number })
| (ExecutionEventBase & {
type: 'VIEW_UPDATE';
step_index: number;
view_descriptor: ViewDescriptor;
})
| (ExecutionEventBase & { type: 'OUTBOX_STAGED'; step_index: number; outbox_entry: OutboxEntry })
| (ExecutionEventBase & { type: 'OUTBOX_COMMITTING'; total_entries: number })
| (ExecutionEventBase & { type: 'OUTBOX_COMMITTED'; step_index: number; receipt: OutboxReceipt })
| (ExecutionEventBase & { type: 'EXECUTION_COMPLETED'; result: ExecutionResult })
| (ExecutionEventBase & { type: 'EXECUTION_FAILED'; error: ExecutionError });

After every state transition, the engine persists the full execution state:

interface ExecutionCheckpoint {
// Identity
execution_id: string;
plan_id: string;
plan_version: string;
// State machine
status: ExecutionStatus;
current_step_index: number;
// Accumulated execution context (JSON-serializable)
context_snapshot: {
intent_inputs: Record<string, any>;
step_outputs: Record<number, Record<string, any>>;
interaction_responses: Record<number, Record<string, any>>;
gate_resolutions: Record<number, GateResolution>;
};
// Step results accumulated so far
step_results: StepResult[];
// Retry state (populated when status is WAITING_RETRY)
retry_info?: {
step_index: number;
retry_count: number;
max_retries: number;
next_retry_at: string; // ISO timestamp
last_error: StepError;
total_retry_wait_ms: number; // Cumulative time spent waiting across all retries
};
// Staged outbox entries (committed only after all steps succeed)
staged_outbox_entries: OutboxEntry[];
// Trace entries accumulated so far
trace_entry_count: number; // Trace is written incrementally, not stored in checkpoint
// Configuration (captured at start, immutable)
execution_mode: ExecutionMode;
environment: string;
user_context: UserContext;
options: ExecutionOptions;
plan: PlanIR; // The full plan (needed for recovery)
// Timing
started_at: string;
last_checkpoint_at: string;
active_duration_ms: number; // Cumulative active execution time
}

Checkpoints are persisted to a Zakardo-managed table in the developer’s database:

-- zakardo_executions table (managed by Zakardo runtime, created on init)
CREATE TABLE zakardo_executions (
execution_id TEXT PRIMARY KEY,
plan_id TEXT NOT NULL,
status TEXT NOT NULL, -- current state machine state
checkpoint JSONB NOT NULL, -- full ExecutionCheckpoint
started_at TIMESTAMP NOT NULL,
last_updated_at TIMESTAMP NOT NULL,
completed_at TIMESTAMP,
-- Indexes for recovery and querying
INDEX idx_status (status),
INDEX idx_plan_id (plan_id),
INDEX idx_started_at (started_at)
);

The checkpoint store uses the developer’s existing database connection (the db service from service configuration). This means:

  • No separate infrastructure for Zakardo state.
  • Checkpoints live alongside the application’s data.
  • The developer’s DB backup strategy covers Zakardo execution state.

For local development, if no database is configured, Zakardo falls back to SQLite (a file-based zakardo.db in the project directory).

Checkpoints are written at every state transition, not after every internal operation:

EventCheckpoint Written?
Execution initializedYes (status: INITIALIZED)
Step completedYes (advance current_step_index, write step output)
Step failed (transient, will retry)Yes (status: WAITING_RETRY, retry_info populated)
Step failed (permanent)Yes (status: PERMANENTLY_FAILED)
Gate triggeredYes (status: WAITING_GATE)
Gate resolvedYes (advance to next phase of step)
Interaction requiredYes (status: WAITING_INTERACTION)
Interaction receivedYes (write response, advance)
All steps doneYes (status: COMMITTING_OUTBOX)
Outbox committedYes (status: COMPLETED)
Mid-capability (during a long-running capability call)No — capabilities are atomic from the engine’s perspective

This means: if the process dies during a capability invocation, the engine will re-execute that step on recovery. Capabilities with side effects must be idempotent (see Section 11).

On startup, the engine runs a recovery sweep:

async function recoverExecutions(): Promise<void> {
// Find all non-terminal executions
const active = await checkpointStore.findByStatus([
'INITIALIZED',
'EXECUTING_STEP', // was mid-step when process died
'WAITING_RETRY',
'WAITING_GATE',
'WAITING_INTERACTION',
'COMMITTING_OUTBOX',
]);
for (const checkpoint of active) {
switch (checkpoint.status) {
case 'INITIALIZED':
case 'EXECUTING_STEP':
// Was mid-step when process died. Re-execute current step.
// Capability must be idempotent (at-least-once semantics).
scheduleAdvance(checkpoint.execution_id, 0); // immediate
break;
case 'WAITING_RETRY':
// Re-schedule the retry timer
const delay = Math.max(
0,
new Date(checkpoint.retry_info.next_retry_at).getTime() - Date.now(),
);
scheduleAdvance(checkpoint.execution_id, delay);
break;
case 'WAITING_GATE':
case 'WAITING_INTERACTION':
// Still waiting. Check if timeout has expired.
if (hasTimedOut(checkpoint)) {
await transitionToPermanentlyFailed(checkpoint, 'TIMEOUT');
}
// Otherwise, do nothing. The response handler will wake us.
break;
case 'COMMITTING_OUTBOX':
// Was mid-outbox-commit. Retry (idempotency keys protect us).
scheduleAdvance(checkpoint.execution_id, 0);
break;
}
}
}

This recovery function runs once on startup. It ensures no execution is left in limbo.

Every execution event is persisted to the zakardo_execution_events table before being emitted to connected clients. This ensures events are never lost — a client that disconnects and reconnects can replay missed events.

CREATE TABLE zakardo_execution_events (
event_id TEXT NOT NULL, -- Monotonic within execution (e.g. "evt_001")
execution_id TEXT NOT NULL REFERENCES zakardo_executions(execution_id),
event_type TEXT NOT NULL, -- e.g. "STEP_COMPLETED"
event_data JSONB NOT NULL, -- Full event payload (JSON-serialized ExecutionEvent)
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
PRIMARY KEY (execution_id, event_id),
INDEX idx_execution_events_type (execution_id, event_type)
);

Event lifecycle:

State transition occurs
│
▼
Build ExecutionEvent (assign next event_id)
│
▼
INSERT INTO zakardo_execution_events ← Durable. Survives crashes.
│
▼
Publish to EventChannel ← Best-effort. For connected clients.

The Event Store is the source of truth for execution progress. The EventChannel is a convenience for live delivery.

Event replay on client connection:

When a client connects to the SSE stream (or polls via the events endpoint), the server:

  1. Reads all stored events for this execution from zakardo_execution_events.
  2. If the client provides a Last-Event-ID, only events after that ID are replayed.
  3. After replay, the server subscribes to the EventChannel for new live events.

Retention: Events are retained for the same duration as the parent execution’s checkpoint. When an execution is archived or purged, its events are purged with it.

The Event Channel is the live delivery mechanism that pushes events from the executor (which may be a different process) to SSE handlers in the API process. It is implemented as an adapter, allowing different backends depending on the deployment model.

interface EventChannel {
/** Publish an event. Called by the executor after storing to Event Store. */
publish(executionId: string, event: ExecutionEvent): Promise<void>;
/** Subscribe to events for an execution. Called by the SSE handler. */
subscribe(executionId: string, handler: (event: ExecutionEvent) => void): Subscription;
}
interface Subscription {
unsubscribe(): void;
}

Built-in implementations:

ImplementationWhen UsedHow It Works
InProcessEventChanneldispatch.mode: 'in-process'Node.js EventEmitter. Zero-cost. Events delivered within the same process.
RedisEventChanneldispatch.mode: 'worker'Redis Pub/Sub. Executor publishes; API process subscribes. Cross-process delivery.

The EventChannel does not guarantee delivery (that’s the Event Store’s job). It is a best-effort push notification for connected clients.

The Execution Engine can run execution work in-process (on the API server’s event loop) or in a separate worker process (via a job queue). This is controlled by configuration, not code changes. The state machine, checkpointing, capability invocation, and event emission code are identical in both modes.

interface Dispatcher {
/**
* Dispatch an execution for processing.
* In in-process mode: runs on the current event loop.
* In worker mode: enqueues a job for a worker to pick up.
*/
dispatch(request: ExecutionRequest): Promise<void>;
/**
* Schedule a delayed dispatch (used for retry timers and recovery).
*/
scheduleDispatch(executionId: string, delayMs: number): Promise<void>;
/**
* Cancel a scheduled dispatch (used when cancelling executions).
*/
cancelScheduledDispatch(executionId: string): Promise<void>;
}
class InProcessDispatcher implements Dispatcher {
async dispatch(request: ExecutionRequest): Promise<void> {
// Run on the event loop — does not block the HTTP response.
// The HTTP handler has already returned 202 Accepted.
setImmediate(() => advanceExecution(request.execution_id));
}
async scheduleDispatch(executionId: string, delayMs: number): Promise<void> {
setTimeout(() => advanceExecution(executionId), delayMs);
}
async cancelScheduledDispatch(executionId: string): Promise<void> {
// Clear the scheduled timeout (tracked by execution_id)
clearTimeout(this.scheduledTimers.get(executionId));
}
}

Characteristics:

  • Zero extra infrastructure. No Redis, no queue, no worker process.
  • Execution runs on the API process event loop via setImmediate.
  • Retry timers use setTimeout (lost on process restart — recovered by §4.4 recovery sweep).
  • Suitable for: development, small teams, low-scale production.

Limitation: All execution work shares CPU and memory with HTTP handling. Heavy plans may degrade API latency. Horizontal scaling pins each execution to one API instance.

class WorkerDispatcher implements Dispatcher {
constructor(private queue: JobQueue) {}
async dispatch(request: ExecutionRequest): Promise<void> {
await this.queue.add('execution:advance', {
executionId: request.execution_id,
});
}
async scheduleDispatch(executionId: string, delayMs: number): Promise<void> {
await this.queue.add(
'execution:advance',
{ executionId },
{
delay: delayMs,
},
);
}
async cancelScheduledDispatch(executionId: string): Promise<void> {
await this.queue.remove('execution:advance', { executionId });
}
}

Worker process:

// worker.ts — runs as a separate Node.js process
import { createZakardoWorker } from '@zakardo/core';
const worker = createZakardoWorker({
config,
capabilities: capabilities.all,
concurrency: 5, // Process up to 5 executions in parallel
});
worker.start();

The worker process:

  1. Consumes jobs from the queue (BullMQ backed by Redis).
  2. Loads the execution checkpoint from the database.
  3. Calls advanceExecution(executionId) — the exact same function the in-process mode uses.
  4. Publishes events via the RedisEventChannel.
  5. Checkpoints after every transition (same as in-process mode).

Characteristics:

  • Execution runs in a dedicated process — API process is never burdened.
  • Workers scale independently: add more workers for more throughput.
  • Retry timers use the queue’s delay feature — survives worker restarts.
  • Suitable for: production, multi-tenant, high-scale.

Requirement: Redis for BullMQ and Pub/Sub.

config/execution.ts
import { defineExecutionConfig } from '@zakardo/core';
export const execution = defineExecutionConfig({
dispatch: {
/*
* 'in-process': Execution runs on the API event loop.
* 'worker': Execution runs in a separate worker process via job queue.
*/
mode: env.ZAKARDO_DISPATCH_MODE ?? 'in-process',
// Worker mode configuration (ignored in in-process mode)
worker: {
queue: {
provider: 'bullmq',
connection: {
host: env.REDIS_HOST ?? 'localhost',
port: Number(env.REDIS_PORT ?? 6379),
},
},
events: {
provider: 'redis-pubsub', // Uses same Redis connection
},
concurrency: Number(env.ZAKARDO_WORKER_CONCURRENCY ?? 5),
},
},
});
AspectIn-ProcessWorker
InfrastructureNoneRedis
Execution locationAPI event loopSeparate process
Retry timerssetTimeout (ephemeral)Queue delay (durable)
Event deliveryEventEmitterRedis Pub/Sub
Horizontal scalingExecutions pinned to one instanceWorkers pick from shared queue
Recovery on crash§4.4 recovery sweep on restart§4.4 sweep + queue re-delivery
Best forDev, prototyping, small appsProduction, multi-tenant, scale

The Execution Engine code (advanceExecution, checkpoint writes, capability invocation, outbox management) is identical in both modes. Only the Dispatcher and EventChannel adapters change. This follows the same adapter pattern used for DataAdapter (Prisma/Drizzle) and AuthAdapter (Better Auth).


Phase 1: INITIALIZATION
Create checkpoint, resolve services, begin Trace, emit EXECUTION_STARTED.
Phase 2: SEQUENTIAL STEP EXECUTION
For step 0, 1, 2, ... N (in order, one at a time):
2a. Per-step policy evaluation
2b. Gate handling (if policy triggers a gate) → may pause
2c. Input resolution (bind inputs from context)
2d. Capability invocation
2e. Output capture + checkpoint
2f. Interaction handling (if capability has Interaction Spec) → may pause
2g. View composition (output views)
2h. Outbox staging (if external effect)
2i. Trace recording
Any step may also → WAITING_RETRY (transient failure)
Phase 3: OUTBOX COMMIT
Commit all staged external effects (with idempotency + retry).
Phase 4: FINALIZATION
Compute plan outputs, finalize Trace, emit completion event.
async function startExecution(request: ExecutionRequest): Promise<string> {
// 1. Generate execution_id
const execution_id = generateId();
// 2. Resolve services for the current environment + mode
const services = serviceRegistry.resolve(request.environment, request.execution_mode);
// 3. Create initial checkpoint
const checkpoint: ExecutionCheckpoint = {
execution_id,
plan_id: request.plan.plan_id,
plan_version: request.plan.version,
status: 'INITIALIZED',
current_step_index: 0,
context_snapshot: {
intent_inputs: request.intent_inputs,
step_outputs: {},
interaction_responses: {},
gate_resolutions: {},
},
step_results: [],
staged_outbox_entries: [],
trace_entry_count: 0,
execution_mode: request.execution_mode,
environment: request.environment,
user_context: request.user_context,
options: request.options ?? defaults,
plan: request.plan,
started_at: new Date().toISOString(),
last_checkpoint_at: new Date().toISOString(),
active_duration_ms: 0,
};
// 4. Persist checkpoint (execution now survives restarts)
await checkpointStore.save(checkpoint);
// 5. Begin Trace
await trace.record(execution_id, 'EXECUTION_STARTED', {
plan_id: request.plan.plan_id,
execution_mode: request.execution_mode,
environment: request.environment,
user_id: request.user_context.user_id,
});
// 6. Emit event to frontend
emit({
type: 'EXECUTION_STARTED',
execution_id,
plan_id: request.plan.plan_id,
total_steps: request.plan.steps.length,
});
// 7. Immediately begin executing first step
await advanceExecution(execution_id);
return execution_id;
}

Service resolution selects the correct service implementations based on the multi-mode service configuration (see System Architecture, Section 6.3):

EnvironmentModedb service resolves to
productionLIVEReal PrismaClient with production DATABASE_URL
productionDRY_RUNReadOnlyWrapper around production PrismaClient
productionSIMULATEDInMemoryDB
developmentLIVEPrismaClient with local dev DB
developmentTRAININGInMemoryDB with fixtures

The core engine loop. Called by every trigger (new execution, step completed, retry timer, gate response, interaction response, process recovery).

async function advanceExecution(execution_id: string): Promise<void> {
const checkpoint = await checkpointStore.load(execution_id);
const step = checkpoint.plan.steps[checkpoint.current_step_index];
const services = serviceRegistry.resolve(checkpoint.environment, checkpoint.execution_mode);
// Build CapabilityContext for this step
const capabilityContext: CapabilityContext = {
execution_id: checkpoint.execution_id,
execution_mode: checkpoint.execution_mode,
environment: checkpoint.environment,
user: checkpoint.user_context,
services,
trace: new TraceWriter(execution_id),
error: createErrorFactory(execution_id),
};
// --- 2a. PER-STEP POLICY EVALUATION ---
const policyResult = await policyEngine.evaluateStep(step, checkpoint);
await trace.record(execution_id, 'POLICY_EVALUATED', {
step_index: step.step_index,
result: policyResult,
});
if (policyResult.action === 'DENY') {
// Permanent failure -- policy blocks this step at runtime
await transitionToPermanentlyFailed(checkpoint, {
code: 'POLICY_DENIED',
category: 'AUTHORIZATION_ERROR',
message: policyResult.reason,
});
return;
}
// --- 2b. GATE HANDLING ---
if (policyResult.action === 'GATE') {
const gateDescriptor = buildGateDescriptor(step, policyResult);
const viewDescriptor = await viewComposition.composeGateView(gateDescriptor);
emit({
type: 'GATE_TRIGGERED',
execution_id,
step_index: step.step_index,
gate: gateDescriptor,
view_descriptor: viewDescriptor,
});
await trace.record(execution_id, 'GATE_TRIGGERED', {
step_index: step.step_index,
gate_type: gateDescriptor.gate_type,
});
// Check v1 auto-DENY rule
if (requiresRoleUserLacks(policyResult, checkpoint.user_context)) {
await transitionToPermanentlyFailed(checkpoint, {
code: 'GATE_AUTO_DENIED',
category: 'AUTHORIZATION_ERROR',
message: 'Gate requires approver role the current user does not have',
});
return;
}
// Pause and wait for gate response
checkpoint.status = 'WAITING_GATE';
await checkpointStore.save(checkpoint);
return; // Execution pauses here. Gate response handler will call advanceExecution.
}
// --- 2c. INPUT RESOLUTION ---
let resolvedInputs: Record<string, any>;
try {
resolvedInputs = resolveInputs(step, checkpoint.context_snapshot);
} catch (bindingError) {
await transitionToPermanentlyFailed(checkpoint, {
code: 'BINDING_ERROR',
category: 'SYSTEM_ERROR',
message: bindingError.message,
});
return;
}
// --- 2d. CAPABILITY INVOCATION ---
emit({
type: 'STEP_STARTED',
execution_id,
step_index: step.step_index,
step_name: step.name,
capability: step.capability,
});
await trace.record(execution_id, 'STEP_STARTED', {
step_index: step.step_index,
inputs: resolvedInputs,
});
const stepStartTime = Date.now();
try {
const output = await withTimeout(
invokeCapability(step, resolvedInputs, capabilityContext),
checkpoint.options.step_timeout_ms ?? 30000,
);
const stepDuration = Date.now() - stepStartTime;
// --- 2e. OUTPUT CAPTURE + CHECKPOINT ---
checkpoint.context_snapshot.step_outputs[step.step_index] = output;
checkpoint.step_results.push({
step_index: step.step_index,
step_name: step.name,
capability: step.capability,
capability_version: step.capability_version,
status: 'COMPLETED',
started_at: new Date(stepStartTime).toISOString(),
completed_at: new Date().toISOString(),
duration_ms: stepDuration,
outputs: output,
retry_count:
checkpoint.retry_info?.step_index === step.step_index
? checkpoint.retry_info.retry_count
: 0,
});
checkpoint.retry_info = undefined; // Clear retry state on success
checkpoint.active_duration_ms += stepDuration;
emit({
type: 'STEP_COMPLETED',
execution_id,
step_index: step.step_index,
outputs: output,
duration_ms: stepDuration,
});
await trace.record(execution_id, 'STEP_COMPLETED', {
step_index: step.step_index,
outputs: output,
duration_ms: stepDuration,
});
// --- 2f. INTERACTION HANDLING ---
if (step.interaction_spec) {
const interactionDescriptor = await viewComposition.composeInteractionView(step, output);
emit({
type: 'INTERACTION_REQUIRED',
execution_id,
step_index: step.step_index,
interaction: interactionDescriptor,
view_descriptor: interactionDescriptor.view_descriptor,
});
await trace.record(execution_id, 'INTERACTION_REQUIRED', { step_index: step.step_index });
checkpoint.status = 'WAITING_INTERACTION';
await checkpointStore.save(checkpoint);
return; // Pause. Interaction response handler will call advanceExecution.
}
// --- 2g. VIEW COMPOSITION ---
const viewDescriptor = await viewComposition.composeOutputView(step, output);
emit({
type: 'VIEW_UPDATE',
execution_id,
step_index: step.step_index,
view_descriptor: viewDescriptor,
});
// --- 2h. OUTBOX STAGING ---
if (needsOutbox(step)) {
const outboxEntry = createOutboxEntry(checkpoint, step, output);
checkpoint.staged_outbox_entries.push(outboxEntry);
emit({
type: 'OUTBOX_STAGED',
execution_id,
step_index: step.step_index,
outbox_entry: outboxEntry,
});
await trace.record(execution_id, 'OUTBOX_STAGED', {
step_index: step.step_index,
outbox_id: outboxEntry.outbox_id,
});
}
// --- ADVANCE TO NEXT STEP OR FINALIZE ---
if (checkpoint.current_step_index < checkpoint.plan.steps.length - 1) {
checkpoint.current_step_index++;
checkpoint.status = 'EXECUTING_STEP';
await checkpointStore.save(checkpoint);
await advanceExecution(execution_id); // Continue immediately with next step
} else {
// All steps done -- proceed to outbox commit
checkpoint.status = 'COMMITTING_OUTBOX';
await checkpointStore.save(checkpoint);
await commitOutbox(checkpoint);
}
} catch (error) {
const stepDuration = Date.now() - stepStartTime;
const categorized = categorizeError(error);
// --- TRANSIENT FAILURE: WAIT AND RETRY ---
if (isTransient(categorized) && retryBudgetRemaining(checkpoint, step)) {
const retryCount = (checkpoint.retry_info?.retry_count ?? 0) + 1;
const delay = computeBackoff(retryCount, step.error_policy);
checkpoint.retry_info = {
step_index: step.step_index,
retry_count: retryCount,
max_retries: step.error_policy?.max_retries ?? defaultMaxRetries(categorized.category),
next_retry_at: new Date(Date.now() + delay).toISOString(),
last_error: categorized,
total_retry_wait_ms: (checkpoint.retry_info?.total_retry_wait_ms ?? 0) + delay,
};
checkpoint.status = 'WAITING_RETRY';
checkpoint.active_duration_ms += stepDuration;
await checkpointStore.save(checkpoint);
emit({
type: 'STEP_FAILED_TRANSIENT',
execution_id,
step_index: step.step_index,
error: categorized,
retry_in_ms: delay,
retry_count: retryCount,
});
await trace.record(execution_id, 'STEP_FAILED_TRANSIENT', {
step_index: step.step_index,
error: categorized,
retry_in_ms: delay,
});
scheduleWakeup(execution_id, delay); // Timer will call advanceExecution
return;
}
// --- PERMANENT FAILURE: STOP ---
checkpoint.active_duration_ms += stepDuration;
await transitionToPermanentlyFailed(checkpoint, categorized);
}
}

After all steps succeed, the engine commits staged external effects:

async function commitOutbox(checkpoint: ExecutionCheckpoint): Promise<void> {
const { execution_id, staged_outbox_entries } = checkpoint;
if (staged_outbox_entries.length === 0) {
// No external effects to commit -- go straight to completion
await finalizeExecution(checkpoint, 'COMPLETED');
return;
}
emit({ type: 'OUTBOX_COMMITTING', execution_id, total_entries: staged_outbox_entries.length });
for (const entry of staged_outbox_entries) {
if (entry.status === 'SUCCEEDED') continue; // Already committed (recovery case)
try {
const receipt = await executeOutboxEntry(entry);
entry.status = 'SUCCEEDED';
entry.receipt = receipt;
entry.committed_at = new Date().toISOString();
emit({ type: 'OUTBOX_COMMITTED', execution_id, step_index: entry.step_index, receipt });
await trace.record(execution_id, 'OUTBOX_COMMITTED', { outbox_id: entry.outbox_id, receipt });
// Checkpoint after each outbox entry (recovery-safe)
await checkpointStore.save(checkpoint);
} catch (error) {
// Outbox commit failure -- retry with backoff
entry.retry_count++;
if (entry.retry_count <= entry.max_retries) {
entry.status = 'RETRYING';
await checkpointStore.save(checkpoint);
const delay = computeBackoff(entry.retry_count);
scheduleWakeup(execution_id, delay);
return; // Will resume on wakeup
} else {
// Outbox entry exhausted retries -- this is a serious issue
// Other entries may have already committed. Log and alert.
entry.status = 'FAILED';
entry.receipt = {
success: false,
error: error.message,
timestamp: new Date().toISOString(),
};
await trace.record(execution_id, 'OUTBOX_FAILED', {
outbox_id: entry.outbox_id,
error: error.message,
});
// Continue with remaining entries -- partial commit is better than full stop
}
}
}
await finalizeExecution(checkpoint, 'COMPLETED');
}

Idempotency is critical here. Each outbox entry has an idempotency_key. If the process dies after committing entry 2 of 5, recovery will re-process entries 1-5, but entries 1-2 will be no-ops because the external system recognizes the idempotency key.

async function finalizeExecution(
checkpoint: ExecutionCheckpoint,
status: 'COMPLETED' | 'PERMANENTLY_FAILED',
error?: ExecutionError,
): Promise<void> {
// 1. Compute plan-level outputs (if completed)
const outputs = status === 'COMPLETED' ? computePlanOutputs(checkpoint) : {};
// 2. Build final result
const result: ExecutionResult = {
execution_id: checkpoint.execution_id,
plan_id: checkpoint.plan_id,
plan_version: checkpoint.plan_version,
status,
execution_mode: checkpoint.execution_mode,
outputs,
trace_id: checkpoint.execution_id, // Trace ID = execution ID
started_at: checkpoint.started_at,
completed_at: new Date().toISOString(),
duration_ms: Date.now() - new Date(checkpoint.started_at).getTime(),
active_duration_ms: checkpoint.active_duration_ms,
step_results: checkpoint.step_results,
error,
};
// 3. Finalize Trace
await trace.record(
checkpoint.execution_id,
status === 'COMPLETED' ? 'EXECUTION_COMPLETED' : 'EXECUTION_FAILED',
{ result },
);
// 4. Update checkpoint to terminal state
checkpoint.status = status;
checkpoint.last_checkpoint_at = new Date().toISOString();
await checkpointStore.save(checkpoint);
// 5. Emit event
if (status === 'COMPLETED') {
emit({ type: 'EXECUTION_COMPLETED', execution_id: checkpoint.execution_id, result });
} else {
emit({ type: 'EXECUTION_FAILED', execution_id: checkpoint.execution_id, error });
}
// 6. If plan was CANDIDATE: record execution result for lifecycle evaluation
if (checkpoint.plan.metadata?.lifecycle_state === 'CANDIDATE') {
await planLibrary.recordExecutionResult(checkpoint.plan_id, result);
}
}

The Execution Context is the accumulated data from step executions. In the durable model, it is part of the checkpoint (persisted as context_snapshot).

interface ContextSnapshot {
// Intent inputs provided by the user (immutable after initialization)
intent_inputs: Record<string, any>;
// Step outputs, keyed by step_index
step_outputs: Record<number, Record<string, any>>;
// Interaction responses, keyed by step_index
interaction_responses: Record<number, Record<string, any>>;
// Gate resolutions, keyed by step_index
gate_resolutions: Record<number, GateResolution>;
}

When a step’s input binding needs to be resolved, the engine reads from the context snapshot:

Binding TypePatternResolution
Prior step output$.steps[N].out.fieldcontext.step_outputs[N].field
Named reference$.named.ref_name.fieldResolve via step’s output_name → step_outputs[index].field
Intent input$.inputs.fieldcontext.intent_inputs.field
Literal value{ "$literal": value }value (used directly)
Transform{ "$transform": { fn, args } }Apply transform function to resolved args
Interaction response$.interactions[N].fieldcontext.interaction_responses[N].field

Because the context snapshot is persisted in the checkpoint, all data must be JSON-serializable. This is guaranteed by design:

  • Capability inputs and outputs conform to JSON Schema (the Type System).
  • Intent inputs are JSON payloads.
  • Interaction responses conform to JSON Schema (the Interaction Spec’s response_schema).
  • Gate resolutions are simple objects.

No Date objects, no Map/Set, no circular references, no functions. All data is plain JSON.


Full execution with real side effects.

  • All capabilities execute with their real (production/environment-appropriate) service implementations.
  • Outbox entries are committed after all steps succeed (real external calls).
  • Gates are enforced (user must respond).
  • Interaction Points are enforced.

Read-only execution for pre-flight validation.

  • READ capabilities: execute normally (real data access).
  • WRITE_INTERNAL capabilities: execute but services resolve to read-only wrappers. Writes are intercepted and logged but not committed.
  • WRITE_EXTERNAL capabilities: capability executes (to produce output data), but the outbox entry is marked as DRY_RUN and never committed. Recorded as “would have done.”
  • DESTRUCTIVE capabilities: blocked at policy level. If one somehow reaches execution, the engine treats it as a permanent failure.

DRY_RUN output includes a dry_run_report:

interface DryRunReport {
would_write: Array<{
step_index: number;
capability: string;
effect_class: 'WRITE_INTERNAL' | 'WRITE_EXTERNAL';
description: string;
data_preview?: Record<string, any>;
}>;
blocked: Array<{
step_index: number;
capability: string;
effect_class: 'DESTRUCTIVE';
reason: string;
}>;
}

No real side effects. Everything is mocked.

  • Services resolve to mock/in-memory implementations (per multi-mode service config).
  • All capabilities execute against mock services.
  • Outbox entries are staged but never committed (discarded at finalization).
  • Gates are auto-approved (no user interaction).
  • Interaction Points use default/sample responses (or are auto-skipped).
  • No checkpointing — SIMULATED runs execute in-memory without persistence (they’re fast, disposable, and don’t need recovery).

A variant of SIMULATED designed for plan training and evaluation.

  • Services resolve to training-specific implementations (fixture databases, known-good datasets).
  • Execution is recorded with enhanced trace detail.
  • Gates and Interaction Points are auto-resolved using training fixtures.
  • Output is compared against expected outputs and scored.
  • No checkpointing — same as SIMULATED, runs are in-memory.

Policy triggers GATE
→ Engine determines gate type
→ A2UI Composition builds Gate A2UI surface
→ Engine emits GATE_TRIGGERED event to frontend
→ Engine checkpoints with status WAITING_GATE
→ Plan pauses (entire plan is blocked -- sequential execution)
→ Frontend renders gate UI
→ User responds (approve, deny, provide input)
→ Frontend sends gate response to engine via HTTP/WebSocket
→ Engine loads checkpoint, validates response
→ Engine records resolution in Trace
→ If approved: checkpoint, resume execution at next phase of current step
→ If denied: transition to PERMANENTLY_FAILED
Gate TypeWhat the User SeesResponse ExpectedOn ApprovalOn Denial
APPROVAL”Step X requires approval. Proceed?”Yes / NoStep executesPlan permanently failed
INPUT”Step X needs additional information”Form dataData written to context; step executesPlan permanently failed
REVIEW”Please review before proceeding”AcknowledgedStep executesPlan permanently failed
LOG_ONLYNothing (not surfaced to user)NoneStep executes immediatelyN/A

In v1, gates are synchronous and single-user:

  • The gate is presented to the current user (the one who initiated the plan).
  • If the gate requires approver roles that the current user does not have: auto-DENY → plan permanently failed.
  • The user must respond within gate_timeout_ms. If they don’t: auto-DENY → plan permanently failed.
  • Gate state survives process restarts. The checkpoint is persisted with status WAITING_GATE. When the user eventually responds (after a restart), the engine loads the checkpoint and resumes.

The “Plans always succeed” principle means the Planner should have detected that the user can’t approve the gate and offered an alternative (Request Pattern). Gates at execution time are for cases where the Planner determined the user CAN approve — they just need to explicitly do so.

v2 will introduce:

  • Async gates: gate response can come hours or days later.
  • Cross-role gates: gate is routed to a user with the required role, not just the current user.
  • Gate queues: pending gates visible in a dashboard for approvers.
  • Gate delegation: current user can delegate the gate to a specific approver.

The durable state machine already supports this — the checkpoint persists the WAITING_GATE state indefinitely. v2 simply adds the routing and notification layer.

interface GateDescriptor {
gate_type: 'APPROVAL' | 'INPUT' | 'REVIEW' | 'LOG_ONLY';
step_index: number;
step_name: string;
capability: string;
effect_class: string;
policy_id: string;
policy_description: string;
context_summary: string;
input_schema?: JSONSchema; // For INPUT gates
review_data?: Record<string, any>; // For REVIEW gates
}

Capability executes and produces output
→ Engine checks if capability has Interaction Spec
→ If yes: A2UI Composition builds Interaction A2UI surface
→ Engine emits INTERACTION_REQUIRED event to frontend
→ Engine checkpoints with status WAITING_INTERACTION
→ Plan pauses (entire plan is blocked -- sequential execution)
→ Frontend renders interaction UI
→ User responds (select, input, confirm, etc.)
→ Frontend sends response to engine
→ Engine loads checkpoint, validates response against Interaction Spec response_schema
→ If valid: write response to context, checkpoint, resume
→ If invalid: request retry from frontend
GateInteraction Point
Triggered byPolicyCapability’s Interaction Spec
PurposeGovernance (approve, review)Functional (select, input, compare)
Data flowGate resolution (approved/denied)User’s response data flows into next steps
Can block plan?Yes (if denied)Yes (if timeout or invalid response)
Skip in SIMULATED?Auto-approveAuto-resolve with defaults/fixtures
Survives restart?Yes (checkpointed)Yes (checkpointed)
interface InteractionDescriptor {
interaction_type: 'SELECT' | 'COMPARE' | 'REVIEW' | 'INPUT' | 'CONFIRM';
step_index: number;
step_name: string;
capability: string;
data: Record<string, any>;
response_schema: JSONSchema;
view_descriptor: ViewDescriptor;
}

The Outbox manages external side effects for capabilities with REMOTE boundary and WRITE_EXTERNAL or DESTRUCTIVE effect class.

In v1, external effects are staged during execution but only committed after all steps succeed. This is a critical design decision:

Step 1: DB write (committed by capability -- internal)
Step 2: Stripe charge (staged in Outbox -- NOT called yet)
Step 3: DB write (committed by capability -- internal)
Step 4: FAILS (permanent)
→ Outbox.discardAll() → Stripe was NEVER called
→ DB writes from steps 1 and 3 ARE committed (capability's responsibility)

If all steps succeed:

Step 1-5: all succeed
→ COMMITTING_OUTBOX phase
→ Stripe is called, webhook is sent, etc.
→ Each outbox entry committed with idempotency key
→ If commit fails: retry with backoff
→ After all committed: COMPLETED

Implication: No step can depend on the result of an external call made by a prior step within the same plan. If step 3 calls Stripe and step 5 needs the Stripe charge ID, this doesn’t work — the Stripe call hasn’t happened yet. In such cases, the capability should be designed to produce the charge ID during the commit phase, or the plan should be split into two plans connected by an async workflow (v2).

  1. Partial execution safety. If any step fails, no external effects have been realized. You don’t need to compensate Stripe for a charge that never happened.
  2. Simpler failure model. The only state that needs manual recovery is internal DB writes from earlier steps. External systems are clean.
  3. Natural fit for sequential execution. Since steps run one at a time and external calls happen at the end, there’s a clean separation between “figure out what to do” and “do it.”
ModeOutbox Behavior
LIVEStage during execution → Commit after all steps succeed → Execute external calls → Record receipts
DRY_RUNStage during execution → Record as “would commit” → Never execute
SIMULATEDStage during execution → Discard at finalization
TRAININGStage during execution → Discard or execute against sandbox (per training config)
interface OutboxEntry {
outbox_id: string;
execution_id: string;
step_index: number;
capability: string;
idempotency_key: string; // execution_id + step_index + capability (default)
action: string; // Human-readable description
payload: Record<string, any>; // Data for the external call
status: 'STAGED' | 'COMMITTING' | 'SUCCEEDED' | 'FAILED' | 'RETRYING' | 'DISCARDED';
created_at: string;
committed_at?: string;
receipt?: OutboxReceipt;
retry_count: number;
max_retries: number; // Default: 5
}
interface OutboxReceipt {
success: boolean;
external_id?: string;
response_data?: Record<string, any>;
error?: string;
timestamp: string;
}

Because the engine checkpoints at state transitions (not mid-capability), a process crash during capability execution means the step will be re-executed on recovery. This is at-least-once semantics.

For READ capabilities, this is harmless.

For WRITE_INTERNAL and WRITE_EXTERNAL capabilities, re-execution could duplicate side effects unless the capability is idempotent.

The engine provides a natural idempotency key to every capability via the CapabilityContext:

// Available in context
const idempotencyKey = `${context.execution_id}:${step.step_index}`;

Capability authors should use this key when performing side effects:

// Example: idempotent database write
export default defineCapability(async (input, context) => {
const db = context.services.db;
const idempotencyKey = `${context.execution_id}:create_order`;
// Upsert with idempotency key -- safe to call twice
const order = await db.order.upsert({
where: { idempotency_key: idempotencyKey },
create: { ...input, idempotency_key: idempotencyKey },
update: {}, // No-op if already exists
});
return { order_id: order.id };
});

Outbox entries have built-in idempotency keys. The Outbox commit phase passes the idempotency key to the external service. If the service supports idempotency (Stripe, most modern APIs), duplicate commits are harmless.

Capabilities with side effects (WRITE_INTERNAL, WRITE_EXTERNAL, DESTRUCTIVE) should be designed for at-least-once execution. This is a capability authoring best practice, not an engine enforcement. The engine provides the idempotency key; the capability author uses it.


Capability errors are categorized to drive automatic behavior:

CategoryNatureMeaningEngine Behavior
TRANSIENTTemporaryNetwork timeout, connection refused, rate limited, service 503Retry with backoff
EXTERNAL_ERRORTemporaryExternal service returned a retryable error (5xx, timeout)Retry with backoff
TIMEOUTTemporaryStep exceeded its timeoutRetry once, then permanent fail
SYSTEM_ERRORAmbiguousInternal error, unhandled exceptionRetry once, then permanent fail
BUSINESS_ERRORPermanentBusiness rule prevents completion (insufficient funds, account locked)Permanent fail immediately
VALIDATION_ERRORPermanentInput data invalid at runtimePermanent fail immediately
AUTHORIZATION_ERRORPermanentRuntime permission check failedPermanent fail immediately

The engine’s core distinction:

  • Transient: The step failed because of a temporary condition. If we wait and try again, it will likely succeed. The plan enters WAITING_RETRY.
  • Permanent: The step failed because of a condition that won’t change by retrying. The plan enters PERMANENTLY_FAILED.

Each step can declare retry configuration. If not declared, defaults apply:

interface StepRetryConfig {
max_retries: number; // How many times to retry
initial_delay_ms: number; // First retry delay
backoff: 'linear' | 'exponential'; // Backoff strategy
max_delay_ms: number; // Cap on backoff delay
}

Defaults by error category:

Categorymax_retriesinitial_delay_msbackoffmax_delay_ms
TRANSIENT31,000exponential30,000
EXTERNAL_ERROR32,000exponential60,000
TIMEOUT15,000linear5,000
SYSTEM_ERROR12,000linear2,000
BUSINESS_ERROR0N/AN/AN/A
VALIDATION_ERROR0N/AN/AN/A
AUTHORIZATION_ERROR0N/AN/AN/A

The retry_budget_ms option (default: 300,000ms = 5 min) limits the total time spent retrying across all steps in a single execution. This prevents a plan from retrying indefinitely if a service is down for a long time.

When the retry budget is exhausted, the next transient failure becomes a permanent failure.

When a step permanently fails:

  1. All staged outbox entries are discarded (external effects never committed).
  2. The Trace records the failure (step, error, retry history).
  3. The checkpoint transitions to PERMANENTLY_FAILED.
  4. The EXECUTION_FAILED event is emitted to the frontend.
  5. Internal DB writes from prior steps remain committed (this is the developer’s responsibility — see Section 11 on idempotency and Section 13 on compensation).

13.1 v1 Model: Deferred Outbox + Observability

Section titled “13.1 v1 Model: Deferred Outbox + Observability”

v1 does not provide automatic compensation (saga/rollback). Instead, it minimizes the need for compensation through architectural design:

Effect Typev1 Compensation Strategy
READNo side effects. Nothing to compensate.
WRITE_INTERNALCapability author’s responsibility. Capabilities manage their own DB transactions. If a later step fails, earlier DB writes remain. The Trace records exactly what was written.
WRITE_EXTERNALDeferred commit via Outbox. External calls only happen after all steps succeed. If any step fails, the outbox is discarded — no external compensation needed.
DESTRUCTIVEPolicy requires gates (user explicitly approves). DRY_RUN first (policy can enforce). The user knows what they’re approving.

Why this works for v1:

  • External effects (the hardest to compensate) are never committed until the plan fully succeeds.
  • Internal DB writes are the developer’s domain — they control the ORM, transactions, and data model.
  • The Trace provides complete data for any manual recovery needed.
  • The validation pipeline (Planner + Validator) catches most issues before execution. Runtime failures are primarily transient (handled by retry).

For plans where internal state consistency is critical, developers can:

  1. Use database transactions within capabilities. A capability can start a transaction and commit/rollback within its own scope.
  2. Design capabilities to be safe. Prefer soft deletes over hard deletes. Use status fields (“pending”, “confirmed”) instead of immediate state changes.
  3. Build recovery plans. If plan A partially fails, the developer can author a “recover from plan A failure” plan that reverses the internal changes.
  4. Use DRY_RUN first. Run the plan in DRY_RUN mode to validate the full flow before running in LIVE mode. Policy can enforce this for certain plan types.

v2 may introduce:

  • Compensation functions: each capability optionally declares a compensate handler.
  • Saga orchestration: the engine runs compensation handlers in reverse order for completed steps when a later step permanently fails.
  • Compensation tracing: every compensation attempt is recorded.
  • Partial compensation: handle the case where compensation itself fails (dead letter queue, manual intervention).

The durable state machine architecture already supports this — compensation is just another phase in the state machine (between PERMANENTLY_FAILED and a new COMPENSATED or COMPENSATION_FAILED terminal state).


Capabilities with USER_ENV boundary execute in the user’s browser, not on the server.

Engine reaches a USER_ENV step
→ Serialize the step's resolved inputs
→ Emit USER_ENV_EXECUTE event to frontend via WebSocket/SSE
→ Engine checkpoints with status WAITING_INTERACTION (reuses interaction wait)
→ Plan pauses
→ Frontend SDK receives the event
→ Frontend SDK loads and executes the capability's client-side implementation
→ Capability executes in the browser (DOM manipulation, local storage, etc.)
→ Frontend SDK sends output back to engine
→ Engine loads checkpoint, writes output to context
→ Execution resumes
  • USER_ENV capabilities don’t call external APIs directly (if they need external data, it comes from prior step outputs).
  • step_timeout_ms applies (accounting for network round-trip).
  • In SIMULATED and TRAINING modes, USER_ENV capabilities are mocked on the server side (no browser interaction).

Each plan execution is fully isolated:

  • Separate execution_id and ExecutionCheckpoint.
  • Separate Trace.
  • Separate staged Outbox.
  • Shared services (but services are designed for concurrent access — they’re DB clients, API clients, etc.).

Multiple plans can execute concurrently for different users or the same user. Each has its own checkpoint. There is no global execution lock.

v1 is sequential. Only one step executes at a time. No concurrent writes to the context. No race conditions between steps.

v2 may introduce parallel execution for steps without data dependencies. The checkpoint model would extend to track multiple in-progress steps.


TimeoutDefaultApplies ToOn Expiry
Overall execution300,000ms (5 min)Active step execution time (excludes wait time)PERMANENTLY_FAILED
Step execution30,000ms (30s)Single capability invocationCategorized as TIMEOUT, retry once then fail
Gate response3,600,000ms (1 hour)Time in WAITING_GATE stateAuto-DENY → PERMANENTLY_FAILED
Interaction response3,600,000ms (1 hour)Time in WAITING_INTERACTION statePERMANENTLY_FAILED
Retry budget300,000ms (5 min)Cumulative time in WAITING_RETRY across all stepsNext transient → permanent

All timeouts are configurable per execution via ExecutionOptions and per plan via Plan IR metadata.

Important: The overall execution timeout counts active execution time only (time spent in capability invocations). Time spent in WAITING_GATE, WAITING_INTERACTION, and WAITING_RETRY does not count against the overall timeout. This prevents governance and infrastructure overhead from killing valid executions. The retry_budget_ms separately caps retry wait time.


Developers can register lifecycle hooks:

config/execution.ts
{
execution: {
hooks: {
beforeExecution?: (request: ExecutionRequest) => Promise<void>;
afterExecution?: (result: ExecutionResult) => Promise<void>;
beforeStep?: (step: PlanStep, checkpoint: ExecutionCheckpoint) => Promise<void>;
afterStep?: (step: PlanStep, result: StepResult, checkpoint: ExecutionCheckpoint) => Promise<void>;
onGateTriggered?: (gate: GateDescriptor, checkpoint: ExecutionCheckpoint) => Promise<void>;
onGateResolved?: (gate: GateDescriptor, resolution: GateResolution) => Promise<void>;
onRetry?: (step: PlanStep, error: StepError, retryCount: number) => Promise<void>;
onPermanentFailure?: (step: PlanStep, error: StepError, checkpoint: ExecutionCheckpoint) => Promise<void>;
}
}
}

Hooks are for observability, logging, and external integrations. They cannot modify the execution flow (they don’t return values the engine uses for decisions). If a hook throws, the error is logged in the Trace but does not affect execution.


Every state transition writes to the Trace:

EventRecorded Data
Execution startedplan_id, execution_mode, environment, user_id, timestamp
Step startedstep_index, capability, resolved inputs, timestamp
Policy evaluatedstep_index, policy results (allow/deny/gate/warn)
Gate triggeredstep_index, gate type, policy_id
Gate resolvedstep_index, resolution (approved/denied), duration
Capability completedstep_index, outputs, duration_ms
Capability failed (transient)step_index, error, retry_in_ms, retry_count
Capability failed (permanent)step_index, error, total_retries
Retry attemptedstep_index, retry_count, previous_error
Outbox stagedstep_index, outbox_id, action
Outbox committedoutbox_id, receipt
Outbox failedoutbox_id, error, retry_count
Interaction requiredstep_index, interaction type
Interaction receivedstep_index, response data
View composedstep_index, view_descriptor type (anchored/ephemeral)
Checkpoint savedstatus, current_step_index, active_duration_ms
Execution completedfinal status, total duration, active duration, output summary

DecisionResolution
Sequential execution in v1Steps execute one at a time, in order. No parallel execution. Pauses block the entire plan. Parallel execution is a future enhancement.
Durable state machineExecution state is checkpointed to persistent storage after every state transition. Executions survive process restarts.
Checkpoint storeZakardo-managed table (zakardo_executions) in the developer’s database. SQLite fallback for local dev.
Process recoveryOn startup, the engine recovers all non-terminal executions: re-executes interrupted steps, re-schedules retry timers, checks timeouts on waiting states.
Transient vs. permanent failureCore distinction. Transient failures (network, rate limit, 503) → automatic retry with backoff. Permanent failures (business rule, auth, validation) → immediate stop.
Retry with backoffTransient failures trigger WAITING_RETRY state with configurable backoff (default: exponential, 3 retries, 30s cap). Retry budget caps total retry time.
Deferred outbox commitExternal effects are staged during execution but only committed after ALL steps succeed. If any step permanently fails, outbox is discarded.
Idempotency requirementCapabilities with side effects should be idempotent. Engine provides execution_id:step_index as natural idempotency key.
No automatic compensation in v1Internal DB writes from completed steps are not rolled back on later failure. External effects are protected by deferred commit. Developer handles internal recovery via Trace data.
Event-driven outputEngine produces events via store-then-emit pattern. Events stored to zakardo_execution_events first, then pushed via EventChannel. Clients consume via SSE or polling endpoint.
Per-step policy evaluationEvery step is policy-checked at execution time. Defense-in-depth behind the Planner’s pre-generation checks.
v1 gates are synchronousGates are presented to the current user. Auto-DENY if user lacks approver roles. Gate state survives restarts (checkpointed).
Mode changes everythingService resolution, outbox behavior, gate handling, interaction handling, and checkpointing all depend on execution mode. SIMULATED and TRAINING skip checkpointing.
Gate/interaction timeout separate from execution timeoutWait time in gates, interactions, and retries does not count against overall execution timeout.
DRY_RUN reportDRY_RUN produces a report of “would have done” writes. Reads execute normally.
Trace everythingEvery state transition, step result, policy check, gate, interaction, retry, outbox entry, and timing is recorded.
Dispatch model (hybrid)Two modes: in-process (zero infrastructure, API event loop) and worker (BullMQ + Redis, separate process). Same state machine code in both. Adapter pattern.
Event StoreAll events persisted to zakardo_execution_events before emitting. Enables client reconnection, catch-up replay, and polling. Events purged with parent execution.
Event Channel adapterInProcessEventChannel (Node.js EventEmitter) for in-process mode. RedisEventChannel (Redis Pub/Sub) for worker mode. Best-effort delivery; Event Store is source of truth.

QuestionDeferred To
What is the exact Trace entry format?Trace System Specification
How does the frontend render A2UI messages and handle events?Frontend Architecture Specification
How does the Plan Library record execution results for lifecycle evaluation?Plan Library and Lifecycle Specification
How are execution hooks registered and managed?Runtime API and SDK Specification
How do training runs compare outputs to expected results?Training and Validation Specification
How does the CLI support local execution for development?CLI and DX Specification
How are execution metrics monitored and alerted on?Trace System Specification
What is the compensation/saga model for v2?Future planning (v2 scope)
How does the checkpoint store migration work?Runtime API and SDK Specification