Trace System
The Trace System is the observability backbone of Zakardo. Every plan execution, every capability invocation, every policy evaluation, every gate resolution, every retry, every error — all are recorded as structured, immutable trace entries. Traces serve auditing, debugging, monitoring, training data, plan promotion evidence, and regression testing. This specification defines the trace format, storage model, query interface, retention, and integration points.
1. Design Goals
Section titled “1. Design Goals”- Everything is traced. No silent execution. Every state transition in the Execution Engine, every Planner decision, every Validator check, every policy evaluation produces a trace entry. If it happened, it’s in the trace.
- Immutable and append-only. Once a trace entry is written, it cannot be modified or deleted (except by retention policy). This is required for audit compliance and replay integrity.
- Structured, not logs. Trace entries are typed, schemaed JSON documents — not free-text log lines. Every entry has a defined type, required fields, and optional fields. They are queryable, filterable, and machine-readable.
- Low overhead. Tracing must not significantly impact execution performance. Target: < 5% overhead on plan execution time for standard trace level.
- Queryable. Developers, operators, and the Training System need to query traces by execution ID, plan ID, user, time range, capability, error type, duration, and more.
- Multi-consumer. Traces are read by: the Frontend (execution progress), the Training System (plan evaluation), the Plan Library (promotion evidence), the CLI (debugging), and Zakardo Cloud (retention, analytics).
2. Trace Model
Section titled “2. Trace Model”2.1 Trace
Section titled “2.1 Trace”A Trace is the complete record of a single plan execution. It is identified by a trace_id which equals the execution_id.
interface Trace { trace_id: string; // Same as execution_id plan_id: string; plan_version: string; execution_mode: ExecutionMode; environment: string; user_id: string; tenant_id?: string; started_at: string; completed_at?: string; duration_ms?: number; active_duration_ms?: number; // Excluding wait time (gates, interactions, retries) status: ExecutionStatus; entry_count: number; entries: TraceEntry[]; // Ordered by timestamp}2.2 Trace Entry
Section titled “2.2 Trace Entry”A Trace Entry is a single event within a trace. Every entry has a common envelope and a typed payload.
interface TraceEntry { // Envelope (common to all entries) entry_id: string; // Unique ID for this entry trace_id: string; // Parent trace timestamp: string; // ISO 8601 sequence: number; // Monotonically increasing within a trace (ordering guarantee) entry_type: TraceEntryType; // Discriminator for the payload step_index?: number; // Which step (if step-specific)
// Payload (varies by entry_type) payload: TracePayload;}
type TraceEntryType = // Execution lifecycle | 'EXECUTION_STARTED' | 'EXECUTION_COMPLETED' | 'EXECUTION_FAILED'
// Step lifecycle | 'STEP_STARTED' | 'STEP_COMPLETED' | 'STEP_FAILED_TRANSIENT' | 'STEP_FAILED_PERMANENT' | 'STEP_RETRY'
// Policy | 'POLICY_EVALUATED'
// Gates | 'GATE_TRIGGERED' | 'GATE_RESOLVED'
// Interactions | 'INTERACTION_REQUIRED' | 'INTERACTION_RECEIVED'
// Outbox | 'OUTBOX_STAGED' | 'OUTBOX_COMMITTED' | 'OUTBOX_FAILED'
// Views | 'VIEW_COMPOSED'
// Checkpoint | 'CHECKPOINT_SAVED'
// Planner (pre-execution) | 'PLANNER_STARTED' | 'PLANNER_COMPLETED' | 'PLANNER_LLM_CALL'
// Validator (pre-execution) | 'VALIDATION_STARTED' | 'VALIDATION_COMPLETED'
// Custom (developer-authored via context.trace) | 'CUSTOM';2.3 Trace Entry Payloads
Section titled “2.3 Trace Entry Payloads”Each entry type has a defined payload schema:
Execution Lifecycle
Section titled “Execution Lifecycle”interface ExecutionStartedPayload { plan_id: string; plan_version: string; execution_mode: ExecutionMode; environment: string; user_id: string; tenant_id?: string; total_steps: number; intent_summary?: string; // Human-readable summary of the intent}
interface ExecutionCompletedPayload { status: 'COMPLETED'; outputs: Record<string, any>; duration_ms: number; active_duration_ms: number; steps_completed: number; steps_retried: number; gates_triggered: number; interactions_completed: number; outbox_entries_committed: number;}
interface ExecutionFailedPayload { status: 'PERMANENTLY_FAILED'; failed_step_index: number; error: { code: string; category: string; message: string; }; steps_completed_before_failure: number; total_retries: number; outbox_entries_discarded: number;}Step Lifecycle
Section titled “Step Lifecycle”interface StepStartedPayload { step_index: number; step_name: string; capability: string; capability_version: string; resolved_inputs: Record<string, any>; // Only at 'verbose' trace level}
interface StepCompletedPayload { step_index: number; step_name: string; capability: string; duration_ms: number; outputs: Record<string, any>; // Only at 'verbose' trace level retry_count: number; // 0 if no retries were needed}
interface StepFailedTransientPayload { step_index: number; capability: string; error: { code: string; category: string; message: string }; retry_count: number; retry_in_ms: number; total_retry_wait_ms: number;}
interface StepFailedPermanentPayload { step_index: number; capability: string; error: { code: string; category: string; message: string }; total_retries: number; total_retry_wait_ms: number;}
interface StepRetryPayload { step_index: number; capability: string; retry_count: number; previous_error: { code: string; category: string; message: string };}Policy
Section titled “Policy”interface PolicyEvaluatedPayload { step_index: number; capability: string; result: 'ALLOW' | 'DENY' | 'GATE' | 'WARN'; policies_checked: number; policy_id?: string; // Which policy triggered the result (if not ALLOW) policy_description?: string; gate_type?: string; // If result is GATE}interface GateTriggeredPayload { step_index: number; gate_type: 'APPROVAL' | 'INPUT' | 'REVIEW' | 'LOG_ONLY'; policy_id: string; capability: string; effect_class: string;}
interface GateResolvedPayload { step_index: number; gate_type: string; resolution: 'APPROVED' | 'DENIED' | 'AUTO_DENIED' | 'TIMED_OUT'; wait_duration_ms: number; responded_by?: string; // user_id of responder}Interactions
Section titled “Interactions”interface InteractionRequiredPayload { step_index: number; interaction_type: 'SELECT' | 'COMPARE' | 'REVIEW' | 'INPUT' | 'CONFIRM'; capability: string;}
interface InteractionReceivedPayload { step_index: number; interaction_type: string; wait_duration_ms: number; response_summary?: string; // Human-readable summary (not the full response data)}Outbox
Section titled “Outbox”interface OutboxStagedPayload { step_index: number; outbox_id: string; capability: string; action: string; // Human-readable description effect_class: string;}
interface OutboxCommittedPayload { outbox_id: string; step_index: number; success: boolean; external_id?: string; duration_ms: number; retry_count: number;}
interface OutboxFailedPayload { outbox_id: string; step_index: number; error: string; retry_count: number; max_retries: number;}Planner
Section titled “Planner”interface PlannerStartedPayload { intent_summary: string; intent_mode: 'expressed' | 'packaged' | 'structured';}
interface PlannerCompletedPayload { response_mode: 'GENERATE' | 'CLARIFY' | 'CONVERSE'; plan_id?: string; // If GENERATE steps_generated?: number; // If GENERATE clarification_question?: string; // If CLARIFY duration_ms: number; llm_calls: number; total_tokens: number; plan_library_hit: boolean; // Was a promoted plan found?}
interface PlannerLLMCallPayload { purpose: string; // 'intent_classification', 'capability_selection', 'input_wiring', etc. provider: string; // 'openai', 'anthropic', etc. model: string; // 'gpt-4', 'claude-3', etc. input_tokens: number; output_tokens: number; duration_ms: number; cost_estimate?: number; // Estimated cost in USD (if available)}Validator
Section titled “Validator”interface ValidationStartedPayload { plan_id: string; plan_version: string; requested_mode: ExecutionMode;}
interface ValidationCompletedPayload { plan_id: string; valid: boolean; error_count: number; warning_count: number; checks_performed: number; duration_ms: number; cache_hit: boolean;}Custom
Section titled “Custom”interface CustomPayload { name: string; // Developer-defined event name data: Record<string, any>; // Developer-defined data}3. Trace Levels
Section titled “3. Trace Levels”The trace_level option controls how much detail is recorded:
| Level | What’s Recorded | Use Case |
|---|---|---|
minimal | Execution lifecycle (start, complete, fail), step lifecycle (start, complete, fail), errors, gates, interactions | Production (low overhead, audit-sufficient) |
standard | Everything in minimal + policy evaluations, outbox events, checkpoints, retries, view compositions, planner/validator events | Default. Good balance of detail and overhead. |
verbose | Everything in standard + resolved inputs for every step, full outputs for every step, LLM prompts/responses, full interaction response data | Development and debugging. Higher overhead. |
3.1 What’s Always Recorded (Regardless of Level)
Section titled “3.1 What’s Always Recorded (Regardless of Level)”- Execution start and completion/failure
- Step start and completion/failure
- Errors (all categories)
- Gate triggers and resolutions
- Interaction requires and receives
These are the minimum for audit compliance. Even minimal captures them.
3.2 Sensitive Data Handling
Section titled “3.2 Sensitive Data Handling”Trace entries may contain sensitive data (user inputs, capability outputs, interaction responses). The Trace System provides:
- Field-level redaction. Capabilities can mark output fields as
sensitivein their schema (using thex-zakardo-sensitive: trueextension). These fields are redacted in trace entries (replaced with"[REDACTED]"). - Input redaction. At
verboselevel, resolved inputs are recorded. Fields marked as sensitive in the capability’sinput_schemaare redacted. - Trace-level configuration. Developers can configure which trace levels are allowed in which environments (e.g.,
verboseonly in development,minimalin production).
{ trace: { level: { production: 'minimal', staging: 'standard', development: 'verbose', }, redact_sensitive: true, // Default: true }}4. Trace Store
Section titled “4. Trace Store”4.1 Storage Model
Section titled “4.1 Storage Model”The Trace Store is an append-only persistent store. In v1, it uses the developer’s database:
-- zakardo_traces tableCREATE TABLE zakardo_traces ( trace_id TEXT PRIMARY KEY, plan_id TEXT NOT NULL, plan_version TEXT NOT NULL, execution_mode TEXT NOT NULL, environment TEXT NOT NULL, user_id TEXT NOT NULL, tenant_id TEXT, status TEXT NOT NULL, started_at TIMESTAMP NOT NULL, completed_at TIMESTAMP, duration_ms INTEGER, active_duration_ms INTEGER, entry_count INTEGER NOT NULL DEFAULT 0,
INDEX idx_plan_id (plan_id), INDEX idx_user_id (user_id), INDEX idx_status (status), INDEX idx_started_at (started_at), INDEX idx_environment (environment));
-- zakardo_trace_entries tableCREATE TABLE zakardo_trace_entries ( entry_id TEXT PRIMARY KEY, trace_id TEXT NOT NULL REFERENCES zakardo_traces(trace_id), sequence INTEGER NOT NULL, entry_type TEXT NOT NULL, step_index INTEGER, timestamp TIMESTAMP NOT NULL, payload JSONB NOT NULL,
INDEX idx_trace_id (trace_id), INDEX idx_entry_type (entry_type), INDEX idx_step_index (trace_id, step_index), UNIQUE (trace_id, sequence));For local development without a database, traces are written to JSON files in a .zakardo/traces/ directory (one file per trace).
4.2 Write Pattern
Section titled “4.2 Write Pattern”Trace entries are written incrementally during execution, not batched at the end:
- Each call to
trace.record(...)creates aTraceEntryand inserts it intozakardo_trace_entries. - The
zakardo_tracesrow is created atEXECUTION_STARTEDand updated atEXECUTION_COMPLETED/EXECUTION_FAILED. - This means partial traces exist for in-progress and failed executions — you can see what happened before the failure.
4.3 Write Performance
Section titled “4.3 Write Performance”To minimize impact on execution:
- Trace writes are asynchronous (fire-and-forget within the same process). The Execution Engine does not
awaittrace writes (except for the final EXECUTION_COMPLETED entry, which is awaited to ensure completeness). - Trace entries are batched in memory and flushed periodically (default: every 100ms or 10 entries, whichever comes first). This reduces database round-trips.
- In SIMULATED and TRAINING modes, traces may be written to memory and flushed at the end (since these executions are fast and disposable).
4.4 Immutability
Section titled “4.4 Immutability”Once written, trace entries are never modified. The Trace Store does not support UPDATE or DELETE on trace entries.
The only exception is retention policy: expired traces can be deleted in bulk (see Section 7).
5. Trace Query Interface
Section titled “5. Trace Query Interface”5.1 Query API
Section titled “5.1 Query API”The Runtime API exposes trace queries:
interface TraceQuery { // Filters (all optional, combined with AND) trace_id?: string; plan_id?: string; user_id?: string; tenant_id?: string; execution_mode?: ExecutionMode; environment?: string; status?: ExecutionStatus | ExecutionStatus[]; started_after?: string; // ISO 8601 started_before?: string; min_duration_ms?: number; max_duration_ms?: number; has_errors?: boolean; has_retries?: boolean; capability?: string; // Any step used this capability entry_type?: TraceEntryType; // Trace contains this entry type
// Pagination limit?: number; // Default: 50, max: 200 offset?: number; sort_by?: 'started_at' | 'duration_ms' | 'entry_count'; sort_order?: 'asc' | 'desc'; // Default: desc}
interface TraceQueryResult { traces: TraceSummary[]; // Trace headers (not full entries) total_count: number; offset: number; limit: number;}
interface TraceSummary { trace_id: string; plan_id: string; plan_version: string; execution_mode: ExecutionMode; environment: string; user_id: string; status: ExecutionStatus; started_at: string; completed_at?: string; duration_ms?: number; active_duration_ms?: number; entry_count: number; error_summary?: string; // If failed: short error description steps_completed: number; retries: number;}5.2 Trace Detail API
Section titled “5.2 Trace Detail API”To fetch the full trace with all entries:
interface TraceDetailQuery { trace_id: string; entry_types?: TraceEntryType[]; // Filter to specific entry types step_index?: number; // Filter to a specific step limit?: number; // Limit entries returned offset?: number;}
interface TraceDetailResult { trace: Trace; // Full trace header entries: TraceEntry[]; // Filtered/paginated entries total_entries: number;}5.3 Common Query Patterns
Section titled “5.3 Common Query Patterns”| Use Case | Query |
|---|---|
| ”Show me the last 10 executions for this plan” | { plan_id: "...", limit: 10, sort_by: 'started_at' } |
| ”Show me all failed executions today” | { status: 'PERMANENTLY_FAILED', started_after: '2026-02-05T00:00:00Z' } |
| ”Show me executions with retries” | { has_retries: true } |
| ”Show me all executions of capability X” | { capability: 'send_email' } |
| ”Show me slow executions (> 10s)“ | { min_duration_ms: 10000 } |
| ”Debug a specific execution” | { trace_id: '...' } (full detail) |
| “Show all LLM calls for an execution” | TraceDetail: { trace_id: '...', entry_types: ['PLANNER_LLM_CALL'] } |
6. Trace Consumers
Section titled “6. Trace Consumers”6.1 Frontend (Real-Time Progress)
Section titled “6.1 Frontend (Real-Time Progress)”During execution, the frontend receives events via WebSocket/SSE (defined in Execution Engine Spec). These events are derived from trace entries. The frontend shows:
- Step-by-step progress (“Running step 3 of 5: send_email”)
- Retry status (“Step 3 failed, retrying in 5s… (attempt 2/3)”)
- Gate/interaction prompts
- Final result
The frontend does not read from the Trace Store directly during execution — it uses the event stream. After execution, the frontend can query the Trace Store for the full trace (e.g., in an execution detail view).
6.2 Training System
Section titled “6.2 Training System”The Training System reads traces to evaluate plan performance:
- Compare actual outputs to expected outputs for training datasets.
- Measure execution timing, error rates, and retry rates.
- Identify plans that consistently fail or require retries.
- Generate plan trust scores based on execution history.
6.3 Plan Library
Section titled “6.3 Plan Library”The Plan Library references traces as evidence for plan promotion:
- A plan’s promotion score may include: success rate across N executions, average duration, error frequency, retry frequency.
- Shadow deployment comparison: compare traces from the promoted plan and the candidate plan running in SIMULATED mode for the same intents.
6.4 CLI (Debugging)
Section titled “6.4 CLI (Debugging)”The CLI provides commands to query and display traces:
$ zakardo trace list --plan send_welcome_email --status failed --limit 5$ zakardo trace show <trace_id>$ zakardo trace show <trace_id> --step 3$ zakardo trace show <trace_id> --type PLANNER_LLM_CALL6.5 Zakardo Cloud (Retention and Analytics)
Section titled “6.5 Zakardo Cloud (Retention and Analytics)”Zakardo Cloud can sync traces from the local Trace Store for:
- Long-term retention (beyond local retention policy).
- Cross-tenant analytics.
- Audit export (for compliance).
- Regression testing (compare traces across plan versions).
7. Retention
Section titled “7. Retention”7.1 Local Retention
Section titled “7.1 Local Retention”By default, the local Trace Store retains traces for a configurable period:
{ trace: { retention: { default_days: 30, // Keep traces for 30 days failed_days: 90, // Keep failed traces longer (for debugging) training_days: 90, // Keep training traces longer }, max_storage_mb: 500, // Optional: cap total trace storage }}A background cleanup job runs periodically (daily by default) and deletes traces older than the retention period.
7.2 Cloud Retention
Section titled “7.2 Cloud Retention”If Zakardo Cloud is enabled, traces are synced to the cloud before local deletion. Cloud retention is configurable per plan (e.g., compliance-sensitive plans may have indefinite retention).
8. Trace-Based Features
Section titled “8. Trace-Based Features”8.1 Execution Replay (v2)
Section titled “8.1 Execution Replay (v2)”In v2, traces enable execution replay: re-running a plan execution with the same inputs and comparing the output to the original trace. This is used for:
- Regression testing after capability updates.
- Debugging: “what changed between this execution and the previous one?”
- Validation: “does the new plan version produce the same outputs as the old one?”
Replay requires that traces at verbose level capture full inputs and outputs.
8.2 Execution Comparison
Section titled “8.2 Execution Comparison”Compare two traces side by side:
- Same plan, different versions (did the update change behavior?).
- Same plan, different users (are there user-specific issues?).
- Promoted plan vs. candidate plan in shadow mode.
8.3 Anomaly Detection (v2)
Section titled “8.3 Anomaly Detection (v2)”In v2, the Trace System can detect anomalies:
- Execution duration significantly longer than historical average.
- Unusual retry rates for a specific capability.
- Sudden increase in failure rates for a plan.
Anomaly detection reads from the Trace Store and emits alerts (via hooks or Zakardo Cloud).
9. Custom Trace Entries
Section titled “9. Custom Trace Entries”Capability authors can write custom trace entries using context.trace:
export default defineCapability(async (input, context) => { const db = context.services.db;
context.trace.record('CUSTOM', { name: 'db_query_started', data: { table: 'customers', query_type: 'findUnique' }, });
const customer = await db.customer.findUnique({ where: { id: input.customer_id } });
context.trace.record('CUSTOM', { name: 'db_query_completed', data: { table: 'customers', found: !!customer, duration_ms: 12 }, });
return { id: customer.id, name: customer.name };});Custom entries appear in the trace alongside system entries, ordered by timestamp. They are useful for:
- Fine-grained debugging within capabilities.
- Performance profiling (database query times, external API latencies).
- Business-level auditing (“checked inventory for SKU X, found 5 units”).
10. TraceWriter Interface
Section titled “10. TraceWriter Interface”The TraceWriter is injected into capabilities via context.trace and used internally by the Execution Engine.
interface TraceWriter { // Record a system trace entry (used by Execution Engine) record(entry_type: TraceEntryType, payload: TracePayload): void;
// Record a custom trace entry (used by capability authors) custom(name: string, data: Record<string, any>): void;
// Get the current trace_id readonly trace_id: string;}record() and custom() are non-blocking (fire-and-forget). They append to an in-memory buffer that is flushed asynchronously to the Trace Store.
11. Summary of Decisions
Section titled “11. Summary of Decisions”| Decision | Resolution |
|---|---|
| Everything is traced | Every state transition, policy check, gate, interaction, retry, outbox event, and error produces a trace entry. |
| Structured entries, not logs | Typed JSON documents with defined schemas per entry type. Machine-readable and queryable. |
| Immutable and append-only | Trace entries are never modified. Deleted only by retention policy. |
| Three trace levels | minimal (audit), standard (default), verbose (debugging). Always-recorded entries regardless of level. |
| Incremental writes | Entries written during execution, not batched at the end. Partial traces exist for in-progress/failed executions. |
| Async, batched writes | Trace writes are fire-and-forget, flushed periodically to minimize execution overhead. |
| Sensitive data redaction | Fields marked x-zakardo-sensitive are redacted in traces. Configurable. |
| Trace ID = Execution ID | One trace per execution. Simple correlation. |
| Local + Cloud storage | v1: local DB tables. Zakardo Cloud: sync for long-term retention and analytics. |
| Configurable retention | Default 30 days, configurable per status (failed/training traces retained longer). |
| Custom entries | Capability authors write custom entries via context.trace.custom(). |
| Planner and Validator tracing | LLM calls, intent classification, validation results are traced. Enables cost tracking and debugging. |
| Replay is v2 | Execution replay from traces requires verbose level. Deferred to v2. |
12. Questions Deferred
Section titled “12. Questions Deferred”| Question | Deferred To |
|---|---|
| How does the frontend render execution progress from the event stream? | Frontend Architecture Specification |
| How does the Training System use traces for plan evaluation and scoring? | Training and Validation Specification |
| How does the Plan Library use traces for promotion evidence? | Plan Library and Lifecycle Specification |
| How does the CLI display traces? | CLI and DX Specification |
| How does Zakardo Cloud sync and retain traces? | PRD — Zakardo Cloud |
| What is the exact trace query API endpoint design? | Runtime API and SDK Specification |
| How does anomaly detection work? | Future planning (v2 scope) |
| How does execution replay work? | Future planning (v2 scope) |