Skip to content

Trace System

The Trace System is the observability backbone of Zakardo. Every plan execution, every capability invocation, every policy evaluation, every gate resolution, every retry, every error — all are recorded as structured, immutable trace entries. Traces serve auditing, debugging, monitoring, training data, plan promotion evidence, and regression testing. This specification defines the trace format, storage model, query interface, retention, and integration points.


  1. Everything is traced. No silent execution. Every state transition in the Execution Engine, every Planner decision, every Validator check, every policy evaluation produces a trace entry. If it happened, it’s in the trace.
  2. Immutable and append-only. Once a trace entry is written, it cannot be modified or deleted (except by retention policy). This is required for audit compliance and replay integrity.
  3. Structured, not logs. Trace entries are typed, schemaed JSON documents — not free-text log lines. Every entry has a defined type, required fields, and optional fields. They are queryable, filterable, and machine-readable.
  4. Low overhead. Tracing must not significantly impact execution performance. Target: < 5% overhead on plan execution time for standard trace level.
  5. Queryable. Developers, operators, and the Training System need to query traces by execution ID, plan ID, user, time range, capability, error type, duration, and more.
  6. Multi-consumer. Traces are read by: the Frontend (execution progress), the Training System (plan evaluation), the Plan Library (promotion evidence), the CLI (debugging), and Zakardo Cloud (retention, analytics).

A Trace is the complete record of a single plan execution. It is identified by a trace_id which equals the execution_id.

interface Trace {
trace_id: string; // Same as execution_id
plan_id: string;
plan_version: string;
execution_mode: ExecutionMode;
environment: string;
user_id: string;
tenant_id?: string;
started_at: string;
completed_at?: string;
duration_ms?: number;
active_duration_ms?: number; // Excluding wait time (gates, interactions, retries)
status: ExecutionStatus;
entry_count: number;
entries: TraceEntry[]; // Ordered by timestamp
}

A Trace Entry is a single event within a trace. Every entry has a common envelope and a typed payload.

interface TraceEntry {
// Envelope (common to all entries)
entry_id: string; // Unique ID for this entry
trace_id: string; // Parent trace
timestamp: string; // ISO 8601
sequence: number; // Monotonically increasing within a trace (ordering guarantee)
entry_type: TraceEntryType; // Discriminator for the payload
step_index?: number; // Which step (if step-specific)
// Payload (varies by entry_type)
payload: TracePayload;
}
type TraceEntryType =
// Execution lifecycle
| 'EXECUTION_STARTED'
| 'EXECUTION_COMPLETED'
| 'EXECUTION_FAILED'
// Step lifecycle
| 'STEP_STARTED'
| 'STEP_COMPLETED'
| 'STEP_FAILED_TRANSIENT'
| 'STEP_FAILED_PERMANENT'
| 'STEP_RETRY'
// Policy
| 'POLICY_EVALUATED'
// Gates
| 'GATE_TRIGGERED'
| 'GATE_RESOLVED'
// Interactions
| 'INTERACTION_REQUIRED'
| 'INTERACTION_RECEIVED'
// Outbox
| 'OUTBOX_STAGED'
| 'OUTBOX_COMMITTED'
| 'OUTBOX_FAILED'
// Views
| 'VIEW_COMPOSED'
// Checkpoint
| 'CHECKPOINT_SAVED'
// Planner (pre-execution)
| 'PLANNER_STARTED'
| 'PLANNER_COMPLETED'
| 'PLANNER_LLM_CALL'
// Validator (pre-execution)
| 'VALIDATION_STARTED'
| 'VALIDATION_COMPLETED'
// Custom (developer-authored via context.trace)
| 'CUSTOM';

Each entry type has a defined payload schema:

interface ExecutionStartedPayload {
plan_id: string;
plan_version: string;
execution_mode: ExecutionMode;
environment: string;
user_id: string;
tenant_id?: string;
total_steps: number;
intent_summary?: string; // Human-readable summary of the intent
}
interface ExecutionCompletedPayload {
status: 'COMPLETED';
outputs: Record<string, any>;
duration_ms: number;
active_duration_ms: number;
steps_completed: number;
steps_retried: number;
gates_triggered: number;
interactions_completed: number;
outbox_entries_committed: number;
}
interface ExecutionFailedPayload {
status: 'PERMANENTLY_FAILED';
failed_step_index: number;
error: {
code: string;
category: string;
message: string;
};
steps_completed_before_failure: number;
total_retries: number;
outbox_entries_discarded: number;
}
interface StepStartedPayload {
step_index: number;
step_name: string;
capability: string;
capability_version: string;
resolved_inputs: Record<string, any>; // Only at 'verbose' trace level
}
interface StepCompletedPayload {
step_index: number;
step_name: string;
capability: string;
duration_ms: number;
outputs: Record<string, any>; // Only at 'verbose' trace level
retry_count: number; // 0 if no retries were needed
}
interface StepFailedTransientPayload {
step_index: number;
capability: string;
error: { code: string; category: string; message: string };
retry_count: number;
retry_in_ms: number;
total_retry_wait_ms: number;
}
interface StepFailedPermanentPayload {
step_index: number;
capability: string;
error: { code: string; category: string; message: string };
total_retries: number;
total_retry_wait_ms: number;
}
interface StepRetryPayload {
step_index: number;
capability: string;
retry_count: number;
previous_error: { code: string; category: string; message: string };
}
interface PolicyEvaluatedPayload {
step_index: number;
capability: string;
result: 'ALLOW' | 'DENY' | 'GATE' | 'WARN';
policies_checked: number;
policy_id?: string; // Which policy triggered the result (if not ALLOW)
policy_description?: string;
gate_type?: string; // If result is GATE
}
interface GateTriggeredPayload {
step_index: number;
gate_type: 'APPROVAL' | 'INPUT' | 'REVIEW' | 'LOG_ONLY';
policy_id: string;
capability: string;
effect_class: string;
}
interface GateResolvedPayload {
step_index: number;
gate_type: string;
resolution: 'APPROVED' | 'DENIED' | 'AUTO_DENIED' | 'TIMED_OUT';
wait_duration_ms: number;
responded_by?: string; // user_id of responder
}
interface InteractionRequiredPayload {
step_index: number;
interaction_type: 'SELECT' | 'COMPARE' | 'REVIEW' | 'INPUT' | 'CONFIRM';
capability: string;
}
interface InteractionReceivedPayload {
step_index: number;
interaction_type: string;
wait_duration_ms: number;
response_summary?: string; // Human-readable summary (not the full response data)
}
interface OutboxStagedPayload {
step_index: number;
outbox_id: string;
capability: string;
action: string; // Human-readable description
effect_class: string;
}
interface OutboxCommittedPayload {
outbox_id: string;
step_index: number;
success: boolean;
external_id?: string;
duration_ms: number;
retry_count: number;
}
interface OutboxFailedPayload {
outbox_id: string;
step_index: number;
error: string;
retry_count: number;
max_retries: number;
}
interface PlannerStartedPayload {
intent_summary: string;
intent_mode: 'expressed' | 'packaged' | 'structured';
}
interface PlannerCompletedPayload {
response_mode: 'GENERATE' | 'CLARIFY' | 'CONVERSE';
plan_id?: string; // If GENERATE
steps_generated?: number; // If GENERATE
clarification_question?: string; // If CLARIFY
duration_ms: number;
llm_calls: number;
total_tokens: number;
plan_library_hit: boolean; // Was a promoted plan found?
}
interface PlannerLLMCallPayload {
purpose: string; // 'intent_classification', 'capability_selection', 'input_wiring', etc.
provider: string; // 'openai', 'anthropic', etc.
model: string; // 'gpt-4', 'claude-3', etc.
input_tokens: number;
output_tokens: number;
duration_ms: number;
cost_estimate?: number; // Estimated cost in USD (if available)
}
interface ValidationStartedPayload {
plan_id: string;
plan_version: string;
requested_mode: ExecutionMode;
}
interface ValidationCompletedPayload {
plan_id: string;
valid: boolean;
error_count: number;
warning_count: number;
checks_performed: number;
duration_ms: number;
cache_hit: boolean;
}
interface CustomPayload {
name: string; // Developer-defined event name
data: Record<string, any>; // Developer-defined data
}

The trace_level option controls how much detail is recorded:

LevelWhat’s RecordedUse Case
minimalExecution lifecycle (start, complete, fail), step lifecycle (start, complete, fail), errors, gates, interactionsProduction (low overhead, audit-sufficient)
standardEverything in minimal + policy evaluations, outbox events, checkpoints, retries, view compositions, planner/validator eventsDefault. Good balance of detail and overhead.
verboseEverything in standard + resolved inputs for every step, full outputs for every step, LLM prompts/responses, full interaction response dataDevelopment and debugging. Higher overhead.

3.1 What’s Always Recorded (Regardless of Level)

Section titled “3.1 What’s Always Recorded (Regardless of Level)”
  • Execution start and completion/failure
  • Step start and completion/failure
  • Errors (all categories)
  • Gate triggers and resolutions
  • Interaction requires and receives

These are the minimum for audit compliance. Even minimal captures them.

Trace entries may contain sensitive data (user inputs, capability outputs, interaction responses). The Trace System provides:

  1. Field-level redaction. Capabilities can mark output fields as sensitive in their schema (using the x-zakardo-sensitive: true extension). These fields are redacted in trace entries (replaced with "[REDACTED]").
  2. Input redaction. At verbose level, resolved inputs are recorded. Fields marked as sensitive in the capability’s input_schema are redacted.
  3. Trace-level configuration. Developers can configure which trace levels are allowed in which environments (e.g., verbose only in development, minimal in production).
config/trace.ts
{
trace: {
level: {
production: 'minimal',
staging: 'standard',
development: 'verbose',
},
redact_sensitive: true, // Default: true
}
}

The Trace Store is an append-only persistent store. In v1, it uses the developer’s database:

-- zakardo_traces table
CREATE TABLE zakardo_traces (
trace_id TEXT PRIMARY KEY,
plan_id TEXT NOT NULL,
plan_version TEXT NOT NULL,
execution_mode TEXT NOT NULL,
environment TEXT NOT NULL,
user_id TEXT NOT NULL,
tenant_id TEXT,
status TEXT NOT NULL,
started_at TIMESTAMP NOT NULL,
completed_at TIMESTAMP,
duration_ms INTEGER,
active_duration_ms INTEGER,
entry_count INTEGER NOT NULL DEFAULT 0,
INDEX idx_plan_id (plan_id),
INDEX idx_user_id (user_id),
INDEX idx_status (status),
INDEX idx_started_at (started_at),
INDEX idx_environment (environment)
);
-- zakardo_trace_entries table
CREATE TABLE zakardo_trace_entries (
entry_id TEXT PRIMARY KEY,
trace_id TEXT NOT NULL REFERENCES zakardo_traces(trace_id),
sequence INTEGER NOT NULL,
entry_type TEXT NOT NULL,
step_index INTEGER,
timestamp TIMESTAMP NOT NULL,
payload JSONB NOT NULL,
INDEX idx_trace_id (trace_id),
INDEX idx_entry_type (entry_type),
INDEX idx_step_index (trace_id, step_index),
UNIQUE (trace_id, sequence)
);

For local development without a database, traces are written to JSON files in a .zakardo/traces/ directory (one file per trace).

Trace entries are written incrementally during execution, not batched at the end:

  1. Each call to trace.record(...) creates a TraceEntry and inserts it into zakardo_trace_entries.
  2. The zakardo_traces row is created at EXECUTION_STARTED and updated at EXECUTION_COMPLETED / EXECUTION_FAILED.
  3. This means partial traces exist for in-progress and failed executions — you can see what happened before the failure.

To minimize impact on execution:

  • Trace writes are asynchronous (fire-and-forget within the same process). The Execution Engine does not await trace writes (except for the final EXECUTION_COMPLETED entry, which is awaited to ensure completeness).
  • Trace entries are batched in memory and flushed periodically (default: every 100ms or 10 entries, whichever comes first). This reduces database round-trips.
  • In SIMULATED and TRAINING modes, traces may be written to memory and flushed at the end (since these executions are fast and disposable).

Once written, trace entries are never modified. The Trace Store does not support UPDATE or DELETE on trace entries.

The only exception is retention policy: expired traces can be deleted in bulk (see Section 7).


The Runtime API exposes trace queries:

interface TraceQuery {
// Filters (all optional, combined with AND)
trace_id?: string;
plan_id?: string;
user_id?: string;
tenant_id?: string;
execution_mode?: ExecutionMode;
environment?: string;
status?: ExecutionStatus | ExecutionStatus[];
started_after?: string; // ISO 8601
started_before?: string;
min_duration_ms?: number;
max_duration_ms?: number;
has_errors?: boolean;
has_retries?: boolean;
capability?: string; // Any step used this capability
entry_type?: TraceEntryType; // Trace contains this entry type
// Pagination
limit?: number; // Default: 50, max: 200
offset?: number;
sort_by?: 'started_at' | 'duration_ms' | 'entry_count';
sort_order?: 'asc' | 'desc'; // Default: desc
}
interface TraceQueryResult {
traces: TraceSummary[]; // Trace headers (not full entries)
total_count: number;
offset: number;
limit: number;
}
interface TraceSummary {
trace_id: string;
plan_id: string;
plan_version: string;
execution_mode: ExecutionMode;
environment: string;
user_id: string;
status: ExecutionStatus;
started_at: string;
completed_at?: string;
duration_ms?: number;
active_duration_ms?: number;
entry_count: number;
error_summary?: string; // If failed: short error description
steps_completed: number;
retries: number;
}

To fetch the full trace with all entries:

interface TraceDetailQuery {
trace_id: string;
entry_types?: TraceEntryType[]; // Filter to specific entry types
step_index?: number; // Filter to a specific step
limit?: number; // Limit entries returned
offset?: number;
}
interface TraceDetailResult {
trace: Trace; // Full trace header
entries: TraceEntry[]; // Filtered/paginated entries
total_entries: number;
}
Use CaseQuery
”Show me the last 10 executions for this plan”{ plan_id: "...", limit: 10, sort_by: 'started_at' }
”Show me all failed executions today”{ status: 'PERMANENTLY_FAILED', started_after: '2026-02-05T00:00:00Z' }
”Show me executions with retries”{ has_retries: true }
”Show me all executions of capability X”{ capability: 'send_email' }
”Show me slow executions (> 10s)“{ min_duration_ms: 10000 }
”Debug a specific execution”{ trace_id: '...' } (full detail)
“Show all LLM calls for an execution”TraceDetail: { trace_id: '...', entry_types: ['PLANNER_LLM_CALL'] }

During execution, the frontend receives events via WebSocket/SSE (defined in Execution Engine Spec). These events are derived from trace entries. The frontend shows:

  • Step-by-step progress (“Running step 3 of 5: send_email”)
  • Retry status (“Step 3 failed, retrying in 5s… (attempt 2/3)”)
  • Gate/interaction prompts
  • Final result

The frontend does not read from the Trace Store directly during execution — it uses the event stream. After execution, the frontend can query the Trace Store for the full trace (e.g., in an execution detail view).

The Training System reads traces to evaluate plan performance:

  • Compare actual outputs to expected outputs for training datasets.
  • Measure execution timing, error rates, and retry rates.
  • Identify plans that consistently fail or require retries.
  • Generate plan trust scores based on execution history.

The Plan Library references traces as evidence for plan promotion:

  • A plan’s promotion score may include: success rate across N executions, average duration, error frequency, retry frequency.
  • Shadow deployment comparison: compare traces from the promoted plan and the candidate plan running in SIMULATED mode for the same intents.

The CLI provides commands to query and display traces:

$ zakardo trace list --plan send_welcome_email --status failed --limit 5
$ zakardo trace show <trace_id>
$ zakardo trace show <trace_id> --step 3
$ zakardo trace show <trace_id> --type PLANNER_LLM_CALL

6.5 Zakardo Cloud (Retention and Analytics)

Section titled “6.5 Zakardo Cloud (Retention and Analytics)”

Zakardo Cloud can sync traces from the local Trace Store for:

  • Long-term retention (beyond local retention policy).
  • Cross-tenant analytics.
  • Audit export (for compliance).
  • Regression testing (compare traces across plan versions).

By default, the local Trace Store retains traces for a configurable period:

config/trace.ts
{
trace: {
retention: {
default_days: 30, // Keep traces for 30 days
failed_days: 90, // Keep failed traces longer (for debugging)
training_days: 90, // Keep training traces longer
},
max_storage_mb: 500, // Optional: cap total trace storage
}
}

A background cleanup job runs periodically (daily by default) and deletes traces older than the retention period.

If Zakardo Cloud is enabled, traces are synced to the cloud before local deletion. Cloud retention is configurable per plan (e.g., compliance-sensitive plans may have indefinite retention).


In v2, traces enable execution replay: re-running a plan execution with the same inputs and comparing the output to the original trace. This is used for:

  • Regression testing after capability updates.
  • Debugging: “what changed between this execution and the previous one?”
  • Validation: “does the new plan version produce the same outputs as the old one?”

Replay requires that traces at verbose level capture full inputs and outputs.

Compare two traces side by side:

  • Same plan, different versions (did the update change behavior?).
  • Same plan, different users (are there user-specific issues?).
  • Promoted plan vs. candidate plan in shadow mode.

In v2, the Trace System can detect anomalies:

  • Execution duration significantly longer than historical average.
  • Unusual retry rates for a specific capability.
  • Sudden increase in failure rates for a plan.

Anomaly detection reads from the Trace Store and emits alerts (via hooks or Zakardo Cloud).


Capability authors can write custom trace entries using context.trace:

export default defineCapability(async (input, context) => {
const db = context.services.db;
context.trace.record('CUSTOM', {
name: 'db_query_started',
data: { table: 'customers', query_type: 'findUnique' },
});
const customer = await db.customer.findUnique({ where: { id: input.customer_id } });
context.trace.record('CUSTOM', {
name: 'db_query_completed',
data: { table: 'customers', found: !!customer, duration_ms: 12 },
});
return { id: customer.id, name: customer.name };
});

Custom entries appear in the trace alongside system entries, ordered by timestamp. They are useful for:

  • Fine-grained debugging within capabilities.
  • Performance profiling (database query times, external API latencies).
  • Business-level auditing (“checked inventory for SKU X, found 5 units”).

The TraceWriter is injected into capabilities via context.trace and used internally by the Execution Engine.

interface TraceWriter {
// Record a system trace entry (used by Execution Engine)
record(entry_type: TraceEntryType, payload: TracePayload): void;
// Record a custom trace entry (used by capability authors)
custom(name: string, data: Record<string, any>): void;
// Get the current trace_id
readonly trace_id: string;
}

record() and custom() are non-blocking (fire-and-forget). They append to an in-memory buffer that is flushed asynchronously to the Trace Store.


DecisionResolution
Everything is tracedEvery state transition, policy check, gate, interaction, retry, outbox event, and error produces a trace entry.
Structured entries, not logsTyped JSON documents with defined schemas per entry type. Machine-readable and queryable.
Immutable and append-onlyTrace entries are never modified. Deleted only by retention policy.
Three trace levelsminimal (audit), standard (default), verbose (debugging). Always-recorded entries regardless of level.
Incremental writesEntries written during execution, not batched at the end. Partial traces exist for in-progress/failed executions.
Async, batched writesTrace writes are fire-and-forget, flushed periodically to minimize execution overhead.
Sensitive data redactionFields marked x-zakardo-sensitive are redacted in traces. Configurable.
Trace ID = Execution IDOne trace per execution. Simple correlation.
Local + Cloud storagev1: local DB tables. Zakardo Cloud: sync for long-term retention and analytics.
Configurable retentionDefault 30 days, configurable per status (failed/training traces retained longer).
Custom entriesCapability authors write custom entries via context.trace.custom().
Planner and Validator tracingLLM calls, intent classification, validation results are traced. Enables cost tracking and debugging.
Replay is v2Execution replay from traces requires verbose level. Deferred to v2.

QuestionDeferred To
How does the frontend render execution progress from the event stream?Frontend Architecture Specification
How does the Training System use traces for plan evaluation and scoring?Training and Validation Specification
How does the Plan Library use traces for promotion evidence?Plan Library and Lifecycle Specification
How does the CLI display traces?CLI and DX Specification
How does Zakardo Cloud sync and retain traces?PRD — Zakardo Cloud
What is the exact trace query API endpoint design?Runtime API and SDK Specification
How does anomaly detection work?Future planning (v2 scope)
How does execution replay work?Future planning (v2 scope)