Skip to content

Training & Validation

The Training System is how Zakardo plans prove themselves. It takes training datasets (intent-response pairs), executes them against the runtime in TRAINING mode, compares actual outputs to expected outputs, scores the results, and drives plan lifecycle transitions — promoting plans that pass, quarantining plans that fail. This specification defines the training dataset format, the training pipeline, result comparison, scoring, regression detection, and integration with the Plan Library and Trace System.


  1. Plans earn trust through execution, not assertion. A plan is not trusted because someone said it works. It’s trusted because it was executed against known inputs and produced correct outputs, repeatedly.
  2. Training is automated. Developers define datasets. The system runs them, scores them, and promotes/quarantines plans without manual intervention (though manual overrides are available).
  3. Regression is detectable. When a capability changes, a plan changes, or a policy changes, training runs reveal regressions — plans that used to pass but no longer do.
  4. Training uses the real pipeline. Training runs go through the full Zakardo pipeline: Planner → Enrichment → Validator → Execution Engine (in TRAINING mode). No shortcuts. If a plan works in training, it works in production (modulo external service differences).
  5. Datasets are portable. Training datasets can be distributed as part of Plan Packs and Capability Packs. A CRM capability pack can include datasets that verify its plans work correctly.

A training dataset is a JSON file containing a collection of test cases. Each test case is an intent paired with expected behavior.

interface TrainingDataset {
dataset_id: string;
name: string;
description?: string;
version: string; // Semver
// What this dataset tests
target_plan_id?: string; // If testing a specific plan
target_capabilities?: string[]; // If testing capabilities (plan is generated)
// Configuration
execution_mode: 'TRAINING'; // Always TRAINING
environment: string; // Which environment config to use (usually 'training' or 'development')
// Test cases
cases: TestCase[];
}
interface TestCase {
case_id: string;
name: string;
description?: string;
// Input
intent: TestIntent;
// Expected behavior
expected: TestExpectation;
// Fixtures (optional)
fixtures?: TestFixtures;
// Tags for filtering
tags?: string[];
}

The intent can be expressed in multiple modes, matching Zakardo’s intent input modes:

type TestIntent =
| { mode: 'expressed'; text: string } // Natural language
| { mode: 'structured'; payload: Record<string, any> } // Pre-parsed structured intent
| { mode: 'plan_direct'; plan_id: string; version: string; inputs: Record<string, any> }; // Skip Planner, test a specific plan directly

The plan_direct mode is important: it lets you test a specific plan version with specific inputs, bypassing the Planner entirely. This isolates plan execution from Planner variability.

interface TestExpectation {
// What should happen?
outcome: 'COMPLETED' | 'PERMANENTLY_FAILED' | 'CLARIFY' | 'CONVERSE';
// For COMPLETED: what should the output look like?
output?: OutputExpectation;
// For PERMANENTLY_FAILED: what error is expected?
expected_error?: {
code?: string;
category?: string;
step_index?: number;
};
// For CLARIFY: what clarification should be asked?
expected_clarification?: {
contains?: string[]; // Clarification text should contain these strings
};
// For CONVERSE: what response is expected?
expected_response?: {
contains?: string[];
};
// Structural expectations (regardless of outcome)
plan_expectations?: {
min_steps?: number;
max_steps?: number;
must_use_capabilities?: string[]; // Plan must include these capabilities
must_not_use_capabilities?: string[];
must_have_gates?: boolean;
};
// Timing expectations
timing?: {
max_duration_ms?: number;
max_active_duration_ms?: number;
};
}

Output comparison is the core of training validation. It supports multiple comparison strategies:

type OutputExpectation =
| { strategy: 'exact'; value: Record<string, any>; }
| { strategy: 'subset'; value: Record<string, any>; }
| { strategy: 'schema'; schema: JSONSchema; }
| { strategy: 'custom'; comparator: string; } // Name of a registered custom comparator
// Examples:
// Exact match: output must equal this exactly
{ strategy: 'exact', value: { status: 'created', id: 'any' } }
// Subset match: output must contain these fields (may have more)
{ strategy: 'subset', value: { status: 'created' } }
// Schema match: output must conform to this JSON Schema
{ strategy: 'schema', schema: { type: 'object', required: ['id', 'status'], properties: { status: { enum: ['created', 'pending'] } } } }
// Custom: developer-defined comparison function
{ strategy: 'custom', comparator: 'compare_customer_output' }

Fixtures prepare the test environment before execution:

interface TestFixtures {
// Seed data for services (loaded into TRAINING mode services)
seed_data?: Record<string, any[]>; // { 'customers': [{...}, {...}], 'orders': [{...}] }
// Mock responses for external services
mock_responses?: Record<string, any>; // { 'stripe_charge': { id: 'ch_test', status: 'succeeded' } }
// User context for this test case
user_context?: {
user_id: string;
tenant_id?: string;
roles: string[];
};
// Gate responses (auto-resolve gates during training)
gate_responses?: Record<number, GateResolution>; // step_index → resolution
// Interaction responses (auto-resolve interactions during training)
interaction_responses?: Record<number, Record<string, any>>; // step_index → response data
}

Training Run triggered (CLI, API, or scheduled)
│
▼
Load Training Dataset
│
▼
For each Test Case (sequentially or in parallel):
│
├── 1. Prepare environment (load fixtures, seed data)
├── 2. Submit intent (via Planner or direct plan execution)
├── 3. Execute plan in TRAINING mode
├── 4. Capture result (output, trace, timing)
├── 5. Compare result to expectation
├── 6. Score the test case (PASS / FAIL / PARTIAL)
└── 7. Record result
│
▼
Aggregate results across all test cases
│
▼
Compute training score
│
▼
Apply lifecycle actions (promote, quarantine, or no action)
│
▼
Emit Training Run Report

For each test case:

  1. Reset TRAINING services. The service configuration for TRAINING mode typically provides in-memory or fixture-based services (see System Architecture, Section 6.3). These are reset between test cases.
  2. Load seed data. If the test case defines fixtures.seed_data, inject it into the TRAINING services.
  3. Configure mock responses. If the test case defines fixtures.mock_responses, configure the mock external services to return those responses.
  4. Set user context. Use the test case’s fixtures.user_context (or a default training user).
  5. Pre-configure gate and interaction responses. If the test case defines auto-responses, register them with the Execution Engine so gates and interactions are auto-resolved during execution.

Depending on the test intent mode:

  • expressed: Submit the natural language text to the Planner. The full pipeline runs: Planner → Enrichment → Validator → Execution Engine.
  • structured: Submit the structured payload to the Planner. Skips intent parsing.
  • plan_direct: Skip the Planner entirely. Load the specified plan from the Plan Library and submit it directly to the Validator → Execution Engine.

The plan executes in TRAINING mode:

  • Services resolve to training implementations (fixture DBs, mock APIs).
  • Gates are auto-resolved using fixture responses (or auto-approved if no fixture).
  • Interactions are auto-resolved using fixture responses (or auto-skipped if no fixture).
  • No checkpointing (TRAINING runs are in-memory, like SIMULATED).
  • Full trace is recorded (at verbose level for training).

After execution, the system compares the actual result against the test expectation:

interface TestCaseResult {
case_id: string;
verdict: 'PASS' | 'FAIL' | 'PARTIAL' | 'ERROR';
score: number; // 0.0 to 1.0
// Outcome check
outcome_match: boolean; // Did the actual outcome match expected?
actual_outcome: string;
expected_outcome: string;
// Output check (if applicable)
output_match?: boolean;
output_diff?: OutputDiff; // Structural diff of actual vs expected
// Plan structure check (if applicable)
plan_match?: boolean;
plan_issues?: string[];
// Timing check (if applicable)
timing_match?: boolean;
actual_duration_ms?: number;
// Error check (if applicable)
error_match?: boolean;
// Full trace reference
trace_id: string;
// Execution details
execution_duration_ms: number;
steps_executed: number;
retries: number;
}

Verdicts:

VerdictMeaning
PASSAll expectations met (outcome, output, plan structure, timing). Score: 1.0.
PARTIALOutcome matched but some secondary expectations failed (e.g., output was close but not exact, timing exceeded). Score: 0.5-0.9 depending on what failed.
FAILOutcome did not match, or critical expectation failed. Score: 0.0-0.4.
ERRORThe test case itself failed to execute (configuration error, fixture error). Score: 0.0.
// Every field must match exactly
// Special values:
// 'any' matches any non-null value
// 'any_string' matches any string
// 'any_number' matches any number
// 'any_array' matches any array
{ strategy: 'exact', value: { id: 'any', status: 'created', name: 'Acme Corp' } }
// Passes if output has id (any value), status === 'created', name === 'Acme Corp'
// Output must contain at least these fields with these values
// Extra fields in the output are ignored
{ strategy: 'subset', value: { status: 'created' } }
// Passes if output.status === 'created', regardless of other fields
// Output must conform to this JSON Schema
{ strategy: 'schema', schema: {
type: 'object',
required: ['id', 'status'],
properties: {
id: { type: 'string' },
status: { enum: ['created', 'pending', 'active'] }
}
}}
config/training.ts
// Developer registers a comparison function
{
training: {
comparators: {
compare_customer_output: (actual, expected, context) => {
// Custom logic
return { match: true, score: 1.0, details: 'Customer fields verified' };
};
}
}
}

interface TrainingRun {
run_id: string;
dataset_id: string;
dataset_version: string;
target_plan_id?: string;
target_plan_version?: string;
started_at: string;
completed_at: string;
duration_ms: number;
// Results
total_cases: number;
passed: number;
failed: number;
partial: number;
errors: number;
pass_rate: number; // 0.0 to 1.0
average_score: number; // 0.0 to 1.0
// Detailed results
case_results: TestCaseResult[];
// Lifecycle actions taken
actions_taken: LifecycleAction[];
// Comparison to previous run (if applicable)
regression?: RegressionReport;
}
interface LifecycleAction {
action: 'promote' | 'quarantine' | 'none';
plan_id: string;
plan_version: string;
reason: string;
}

Training runs can be triggered by:

TriggerDescription
CLI commandzakardo training run <dataset_id>
API callPOST /api/training/runs
ScheduleCron-like schedule in config/training.ts (e.g., “run nightly”)
Capability changeWhen a capability used by a plan has a version update, auto-run affected datasets
Policy changeWhen a policy changes, auto-run datasets for affected plans
Plan submissionWhen a new plan enters the library as CANDIDATE, auto-run its associated dataset (if any)
config/training.ts
{
training: {
// Auto-triggers
auto_run_on_capability_change: true,
auto_run_on_policy_change: true,
auto_run_on_plan_submit: true,
// Scheduling
schedule: '0 2 * * *', // Run all datasets at 2 AM daily (cron syntax)
// Execution
parallel_cases: 5, // Run up to 5 test cases in parallel
timeout_per_case_ms: 60000, // 60s per test case
// Lifecycle actions
auto_promote: true, // Automatically promote plans that meet criteria
auto_quarantine: true, // Automatically quarantine plans that fail
// Custom comparators
comparators: { ... },
}
}

A regression occurs when a test case that previously passed now fails. Regressions are the primary signal that something changed (capability update, policy change, data change) and broke an existing behavior.

After each training run, the system compares results against the last successful run of the same dataset:

interface RegressionReport {
previous_run_id: string;
previous_pass_rate: number;
current_pass_rate: number;
delta: number; // Positive = improvement, negative = regression
regressions: Regression[]; // Cases that went from PASS to FAIL
improvements: Improvement[]; // Cases that went from FAIL to PASS
new_failures: string[]; // New cases that failed (not in previous run)
}
interface Regression {
case_id: string;
case_name: string;
previous_verdict: 'PASS';
current_verdict: 'FAIL' | 'PARTIAL';
likely_cause?: string; // Heuristic: "capability X was updated", "policy Y changed"
trace_id: string; // Trace of the failing run (for debugging)
}

The system attempts to identify why a regression occurred by checking:

  1. Capability version changes. If any capability used by the plan changed version since the last passing run, it’s flagged as a likely cause.
  2. Policy changes. If any applicable policy was modified, it’s flagged.
  3. Dataset changes. If the dataset itself changed (new version), the regression may be in the dataset, not the plan.
  4. Service configuration changes. If the TRAINING service config changed, it’s flagged.

This is heuristic, not definitive. The trace provides the detailed evidence for debugging.


6. Training and Plan Lifecycle Integration

Section titled “6. Training and Plan Lifecycle Integration”
Training run completes with pass_rate >= promotion threshold
AND target plan is CANDIDATE or SHADOW
AND auto_promote is enabled
→ Plan Library: transition plan to PROMOTED (or SHADOW if criteria not fully met)
→ Record lifecycle action in training run report
Training run completes with pass_rate < quarantine threshold
AND target plan is PROMOTED or SHADOW
AND auto_quarantine is enabled
→ Plan Library: transition plan to QUARANTINED
→ Record lifecycle action in training run report
→ Emit alert (via hooks)
config/training.ts
{
training: {
promotion_threshold: 0.95, // 95% pass rate to promote
quarantine_threshold: 0.50, // Below 50% pass rate to quarantine
}
}

After each training run, the plan’s trust_signals.training_pass_rate is updated. This feeds into the trust score calculation (see Plan Library Specification, Section 5).


Datasets are JSON files:

{
"dataset_id": "ds_send_welcome_email",
"name": "Send Welcome Email - Core Scenarios",
"version": "1.0.0",
"target_plan_id": "plan_send_welcome_email",
"execution_mode": "TRAINING",
"environment": "development",
"cases": [
{
"case_id": "happy_path",
"name": "New customer receives welcome email",
"intent": {
"mode": "plan_direct",
"plan_id": "plan_send_welcome_email",
"version": "2.0.0",
"inputs": { "customer_id": "cust_001" }
},
"fixtures": {
"seed_data": {
"customers": [
{
"id": "cust_001",
"name": "Jane Doe",
"email": "[email protected]",
"status": "active"
}
]
},
"mock_responses": {
"send_email": { "message_id": "msg_test_001", "status": "sent" }
}
},
"expected": {
"outcome": "COMPLETED",
"output": {
"strategy": "subset",
"value": { "email_sent": true, "message_id": "msg_test_001" }
},
"plan_expectations": {
"must_use_capabilities": ["get_customer", "render_template", "send_email"]
},
"timing": { "max_duration_ms": 5000 }
}
},
{
"case_id": "customer_not_found",
"name": "Non-existent customer returns error",
"intent": {
"mode": "plan_direct",
"plan_id": "plan_send_welcome_email",
"version": "2.0.0",
"inputs": { "customer_id": "cust_999" }
},
"fixtures": {
"seed_data": { "customers": [] }
},
"expected": {
"outcome": "PERMANENTLY_FAILED",
"expected_error": {
"code": "CUSTOMER_NOT_FOUND",
"category": "BUSINESS_ERROR",
"step_index": 0
}
}
},
{
"case_id": "planner_intent_test",
"name": "Natural language intent resolves to correct plan",
"intent": {
"mode": "expressed",
"text": "send a welcome email to customer Jane Doe"
},
"fixtures": {
"seed_data": {
"customers": [
{
"id": "cust_001",
"name": "Jane Doe",
"email": "[email protected]",
"status": "active"
}
]
}
},
"expected": {
"outcome": "COMPLETED",
"plan_expectations": {
"must_use_capabilities": ["get_customer", "send_email"]
}
},
"tags": ["planner", "intent_resolution"]
}
]
}

Datasets live alongside the plans they test:

my-crm-app/
zakardo/
plans/
send_welcome_email.plan.json
training/
send_welcome_email.dataset.json
create_customer.dataset.json
capabilities/
...

Or within a Capability Pack / Plan Pack:

@acme/crm-pack/
capabilities/
...
plans/
...
training/
...
pack.json

Developers can generate a training dataset from an existing successful trace:

$ zakardo training generate-dataset --trace <trace_id> --output ./training/my_dataset.json

This creates a dataset with:

  • Intent from the trace’s original intent.
  • Expected output from the trace’s actual output.
  • Default strategy: subset match.

The developer can then refine the generated dataset (add more cases, adjust expectations, add fixtures).


-- zakardo_training_runs table
CREATE TABLE zakardo_training_runs (
run_id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL,
dataset_version TEXT NOT NULL,
target_plan_id TEXT,
target_plan_version TEXT,
started_at TIMESTAMP NOT NULL,
completed_at TIMESTAMP,
duration_ms INTEGER,
total_cases INTEGER NOT NULL,
passed INTEGER NOT NULL DEFAULT 0,
failed INTEGER NOT NULL DEFAULT 0,
partial INTEGER NOT NULL DEFAULT 0,
errors INTEGER NOT NULL DEFAULT 0,
pass_rate REAL,
average_score REAL,
case_results JSONB NOT NULL DEFAULT '[]',
actions_taken JSONB NOT NULL DEFAULT '[]',
regression JSONB,
INDEX idx_dataset (dataset_id),
INDEX idx_plan (target_plan_id),
INDEX idx_started (started_at)
);

DecisionResolution
Plans earn trust through executionPromotion is driven by training results + execution history, not manual assertion.
Training uses the real pipelineTRAINING mode goes through Planner → Enrichment → Validator → Execution Engine. No shortcuts.
Three intent modes for test casesexpressed (natural language, full pipeline), structured (pre-parsed), plan_direct (skip Planner, test plan in isolation).
Four output comparison strategiesexact, subset, schema, custom comparator.
Four verdictsPASS (1.0), PARTIAL (0.5-0.9), FAIL (0.0-0.4), ERROR (0.0).
Auto-promotion and auto-quarantineTraining results can automatically trigger lifecycle transitions. Configurable thresholds.
Regression detectionCompare current run to previous run. Flag cases that went from PASS to FAIL. Heuristic likely-cause analysis.
Fixtures for determinismSeed data, mock responses, auto-gate/interaction responses ensure repeatable test cases.
Dataset portabilityDatasets distributed in Plan Packs and Capability Packs.
Auto-generate datasets from tracesCLI command creates a starter dataset from a successful execution trace.
Auto-trigger on changesTraining runs auto-trigger on capability version change, policy change, or plan submission.
TRAINING mode = in-memoryNo checkpointing, no outbox commits. Fixture services. Full trace at verbose level.

QuestionDeferred To
What is the exact CLI interface for training commands?CLI and DX Specification
How does the Runtime API expose training runs?Runtime API and SDK Specification
How does Zakardo Cloud aggregate training results across tenants?PRD — Zakardo Cloud
How are datasets and training results displayed in the frontend?Frontend Architecture Specification
How do Plan Packs and Capability Packs bundle datasets?Registry and Marketplace Specification