Training & Validation
The Training System is how Zakardo plans prove themselves. It takes training datasets (intent-response pairs), executes them against the runtime in TRAINING mode, compares actual outputs to expected outputs, scores the results, and drives plan lifecycle transitions — promoting plans that pass, quarantining plans that fail. This specification defines the training dataset format, the training pipeline, result comparison, scoring, regression detection, and integration with the Plan Library and Trace System.
1. Design Goals
Section titled “1. Design Goals”- Plans earn trust through execution, not assertion. A plan is not trusted because someone said it works. It’s trusted because it was executed against known inputs and produced correct outputs, repeatedly.
- Training is automated. Developers define datasets. The system runs them, scores them, and promotes/quarantines plans without manual intervention (though manual overrides are available).
- Regression is detectable. When a capability changes, a plan changes, or a policy changes, training runs reveal regressions — plans that used to pass but no longer do.
- Training uses the real pipeline. Training runs go through the full Zakardo pipeline: Planner → Enrichment → Validator → Execution Engine (in TRAINING mode). No shortcuts. If a plan works in training, it works in production (modulo external service differences).
- Datasets are portable. Training datasets can be distributed as part of Plan Packs and Capability Packs. A CRM capability pack can include datasets that verify its plans work correctly.
2. Training Dataset
Section titled “2. Training Dataset”2.1 Dataset Structure
Section titled “2.1 Dataset Structure”A training dataset is a JSON file containing a collection of test cases. Each test case is an intent paired with expected behavior.
interface TrainingDataset { dataset_id: string; name: string; description?: string; version: string; // Semver
// What this dataset tests target_plan_id?: string; // If testing a specific plan target_capabilities?: string[]; // If testing capabilities (plan is generated)
// Configuration execution_mode: 'TRAINING'; // Always TRAINING environment: string; // Which environment config to use (usually 'training' or 'development')
// Test cases cases: TestCase[];}2.2 Test Case
Section titled “2.2 Test Case”interface TestCase { case_id: string; name: string; description?: string;
// Input intent: TestIntent;
// Expected behavior expected: TestExpectation;
// Fixtures (optional) fixtures?: TestFixtures;
// Tags for filtering tags?: string[];}2.3 Test Intent
Section titled “2.3 Test Intent”The intent can be expressed in multiple modes, matching Zakardo’s intent input modes:
type TestIntent = | { mode: 'expressed'; text: string } // Natural language | { mode: 'structured'; payload: Record<string, any> } // Pre-parsed structured intent | { mode: 'plan_direct'; plan_id: string; version: string; inputs: Record<string, any> }; // Skip Planner, test a specific plan directlyThe plan_direct mode is important: it lets you test a specific plan version with specific inputs, bypassing the Planner entirely. This isolates plan execution from Planner variability.
2.4 Test Expectation
Section titled “2.4 Test Expectation”interface TestExpectation { // What should happen? outcome: 'COMPLETED' | 'PERMANENTLY_FAILED' | 'CLARIFY' | 'CONVERSE';
// For COMPLETED: what should the output look like? output?: OutputExpectation;
// For PERMANENTLY_FAILED: what error is expected? expected_error?: { code?: string; category?: string; step_index?: number; };
// For CLARIFY: what clarification should be asked? expected_clarification?: { contains?: string[]; // Clarification text should contain these strings };
// For CONVERSE: what response is expected? expected_response?: { contains?: string[]; };
// Structural expectations (regardless of outcome) plan_expectations?: { min_steps?: number; max_steps?: number; must_use_capabilities?: string[]; // Plan must include these capabilities must_not_use_capabilities?: string[]; must_have_gates?: boolean; };
// Timing expectations timing?: { max_duration_ms?: number; max_active_duration_ms?: number; };}2.5 Output Expectation
Section titled “2.5 Output Expectation”Output comparison is the core of training validation. It supports multiple comparison strategies:
type OutputExpectation = | { strategy: 'exact'; value: Record<string, any>; } | { strategy: 'subset'; value: Record<string, any>; } | { strategy: 'schema'; schema: JSONSchema; } | { strategy: 'custom'; comparator: string; } // Name of a registered custom comparator
// Examples:
// Exact match: output must equal this exactly{ strategy: 'exact', value: { status: 'created', id: 'any' } }
// Subset match: output must contain these fields (may have more){ strategy: 'subset', value: { status: 'created' } }
// Schema match: output must conform to this JSON Schema{ strategy: 'schema', schema: { type: 'object', required: ['id', 'status'], properties: { status: { enum: ['created', 'pending'] } } } }
// Custom: developer-defined comparison function{ strategy: 'custom', comparator: 'compare_customer_output' }2.6 Test Fixtures
Section titled “2.6 Test Fixtures”Fixtures prepare the test environment before execution:
interface TestFixtures { // Seed data for services (loaded into TRAINING mode services) seed_data?: Record<string, any[]>; // { 'customers': [{...}, {...}], 'orders': [{...}] }
// Mock responses for external services mock_responses?: Record<string, any>; // { 'stripe_charge': { id: 'ch_test', status: 'succeeded' } }
// User context for this test case user_context?: { user_id: string; tenant_id?: string; roles: string[]; };
// Gate responses (auto-resolve gates during training) gate_responses?: Record<number, GateResolution>; // step_index → resolution
// Interaction responses (auto-resolve interactions during training) interaction_responses?: Record<number, Record<string, any>>; // step_index → response data}3. Training Pipeline
Section titled “3. Training Pipeline”3.1 Pipeline Overview
Section titled “3.1 Pipeline Overview”Training Run triggered (CLI, API, or scheduled) │ ▼Load Training Dataset │ ▼For each Test Case (sequentially or in parallel): │ ├── 1. Prepare environment (load fixtures, seed data) ├── 2. Submit intent (via Planner or direct plan execution) ├── 3. Execute plan in TRAINING mode ├── 4. Capture result (output, trace, timing) ├── 5. Compare result to expectation ├── 6. Score the test case (PASS / FAIL / PARTIAL) └── 7. Record result │ ▼Aggregate results across all test cases │ ▼Compute training score │ ▼Apply lifecycle actions (promote, quarantine, or no action) │ ▼Emit Training Run Report3.2 Phase 1: Environment Preparation
Section titled “3.2 Phase 1: Environment Preparation”For each test case:
- Reset TRAINING services. The service configuration for TRAINING mode typically provides in-memory or fixture-based services (see System Architecture, Section 6.3). These are reset between test cases.
- Load seed data. If the test case defines
fixtures.seed_data, inject it into the TRAINING services. - Configure mock responses. If the test case defines
fixtures.mock_responses, configure the mock external services to return those responses. - Set user context. Use the test case’s
fixtures.user_context(or a default training user). - Pre-configure gate and interaction responses. If the test case defines auto-responses, register them with the Execution Engine so gates and interactions are auto-resolved during execution.
3.3 Phase 2: Intent Submission
Section titled “3.3 Phase 2: Intent Submission”Depending on the test intent mode:
expressed: Submit the natural language text to the Planner. The full pipeline runs: Planner → Enrichment → Validator → Execution Engine.structured: Submit the structured payload to the Planner. Skips intent parsing.plan_direct: Skip the Planner entirely. Load the specified plan from the Plan Library and submit it directly to the Validator → Execution Engine.
3.4 Phase 3: Execution
Section titled “3.4 Phase 3: Execution”The plan executes in TRAINING mode:
- Services resolve to training implementations (fixture DBs, mock APIs).
- Gates are auto-resolved using fixture responses (or auto-approved if no fixture).
- Interactions are auto-resolved using fixture responses (or auto-skipped if no fixture).
- No checkpointing (TRAINING runs are in-memory, like SIMULATED).
- Full trace is recorded (at
verboselevel for training).
3.5 Phase 4: Result Comparison
Section titled “3.5 Phase 4: Result Comparison”After execution, the system compares the actual result against the test expectation:
interface TestCaseResult { case_id: string; verdict: 'PASS' | 'FAIL' | 'PARTIAL' | 'ERROR'; score: number; // 0.0 to 1.0
// Outcome check outcome_match: boolean; // Did the actual outcome match expected? actual_outcome: string; expected_outcome: string;
// Output check (if applicable) output_match?: boolean; output_diff?: OutputDiff; // Structural diff of actual vs expected
// Plan structure check (if applicable) plan_match?: boolean; plan_issues?: string[];
// Timing check (if applicable) timing_match?: boolean; actual_duration_ms?: number;
// Error check (if applicable) error_match?: boolean;
// Full trace reference trace_id: string;
// Execution details execution_duration_ms: number; steps_executed: number; retries: number;}Verdicts:
| Verdict | Meaning |
|---|---|
PASS | All expectations met (outcome, output, plan structure, timing). Score: 1.0. |
PARTIAL | Outcome matched but some secondary expectations failed (e.g., output was close but not exact, timing exceeded). Score: 0.5-0.9 depending on what failed. |
FAIL | Outcome did not match, or critical expectation failed. Score: 0.0-0.4. |
ERROR | The test case itself failed to execute (configuration error, fixture error). Score: 0.0. |
3.6 Phase 5: Output Comparison Strategies
Section titled “3.6 Phase 5: Output Comparison Strategies”Exact Match
Section titled “Exact Match”// Every field must match exactly// Special values:// 'any' matches any non-null value// 'any_string' matches any string// 'any_number' matches any number// 'any_array' matches any array
{ strategy: 'exact', value: { id: 'any', status: 'created', name: 'Acme Corp' } }// Passes if output has id (any value), status === 'created', name === 'Acme Corp'Subset Match
Section titled “Subset Match”// Output must contain at least these fields with these values// Extra fields in the output are ignored
{ strategy: 'subset', value: { status: 'created' } }// Passes if output.status === 'created', regardless of other fieldsSchema Match
Section titled “Schema Match”// Output must conform to this JSON Schema{ strategy: 'schema', schema: { type: 'object', required: ['id', 'status'], properties: { id: { type: 'string' }, status: { enum: ['created', 'pending', 'active'] } }}}Custom Comparator
Section titled “Custom Comparator”// Developer registers a comparison function{ training: { comparators: { compare_customer_output: (actual, expected, context) => { // Custom logic return { match: true, score: 1.0, details: 'Customer fields verified' }; }; } }}4. Training Run
Section titled “4. Training Run”4.1 Training Run Record
Section titled “4.1 Training Run Record”interface TrainingRun { run_id: string; dataset_id: string; dataset_version: string; target_plan_id?: string; target_plan_version?: string; started_at: string; completed_at: string; duration_ms: number;
// Results total_cases: number; passed: number; failed: number; partial: number; errors: number; pass_rate: number; // 0.0 to 1.0 average_score: number; // 0.0 to 1.0
// Detailed results case_results: TestCaseResult[];
// Lifecycle actions taken actions_taken: LifecycleAction[];
// Comparison to previous run (if applicable) regression?: RegressionReport;}
interface LifecycleAction { action: 'promote' | 'quarantine' | 'none'; plan_id: string; plan_version: string; reason: string;}4.2 Triggering Training Runs
Section titled “4.2 Triggering Training Runs”Training runs can be triggered by:
| Trigger | Description |
|---|---|
| CLI command | zakardo training run <dataset_id> |
| API call | POST /api/training/runs |
| Schedule | Cron-like schedule in config/training.ts (e.g., “run nightly”) |
| Capability change | When a capability used by a plan has a version update, auto-run affected datasets |
| Policy change | When a policy changes, auto-run datasets for affected plans |
| Plan submission | When a new plan enters the library as CANDIDATE, auto-run its associated dataset (if any) |
4.3 Configuration
Section titled “4.3 Configuration”{ training: { // Auto-triggers auto_run_on_capability_change: true, auto_run_on_policy_change: true, auto_run_on_plan_submit: true,
// Scheduling schedule: '0 2 * * *', // Run all datasets at 2 AM daily (cron syntax)
// Execution parallel_cases: 5, // Run up to 5 test cases in parallel timeout_per_case_ms: 60000, // 60s per test case
// Lifecycle actions auto_promote: true, // Automatically promote plans that meet criteria auto_quarantine: true, // Automatically quarantine plans that fail
// Custom comparators comparators: { ... }, }}5. Regression Detection
Section titled “5. Regression Detection”5.1 What Is a Regression?
Section titled “5.1 What Is a Regression?”A regression occurs when a test case that previously passed now fails. Regressions are the primary signal that something changed (capability update, policy change, data change) and broke an existing behavior.
5.2 Regression Report
Section titled “5.2 Regression Report”After each training run, the system compares results against the last successful run of the same dataset:
interface RegressionReport { previous_run_id: string; previous_pass_rate: number; current_pass_rate: number; delta: number; // Positive = improvement, negative = regression
regressions: Regression[]; // Cases that went from PASS to FAIL improvements: Improvement[]; // Cases that went from FAIL to PASS new_failures: string[]; // New cases that failed (not in previous run)}
interface Regression { case_id: string; case_name: string; previous_verdict: 'PASS'; current_verdict: 'FAIL' | 'PARTIAL'; likely_cause?: string; // Heuristic: "capability X was updated", "policy Y changed" trace_id: string; // Trace of the failing run (for debugging)}5.3 Likely Cause Heuristic
Section titled “5.3 Likely Cause Heuristic”The system attempts to identify why a regression occurred by checking:
- Capability version changes. If any capability used by the plan changed version since the last passing run, it’s flagged as a likely cause.
- Policy changes. If any applicable policy was modified, it’s flagged.
- Dataset changes. If the dataset itself changed (new version), the regression may be in the dataset, not the plan.
- Service configuration changes. If the TRAINING service config changed, it’s flagged.
This is heuristic, not definitive. The trace provides the detailed evidence for debugging.
6. Training and Plan Lifecycle Integration
Section titled “6. Training and Plan Lifecycle Integration”6.1 Auto-Promotion Flow
Section titled “6.1 Auto-Promotion Flow”Training run completes with pass_rate >= promotion threshold AND target plan is CANDIDATE or SHADOW AND auto_promote is enabled → Plan Library: transition plan to PROMOTED (or SHADOW if criteria not fully met) → Record lifecycle action in training run report6.2 Auto-Quarantine Flow
Section titled “6.2 Auto-Quarantine Flow”Training run completes with pass_rate < quarantine threshold AND target plan is PROMOTED or SHADOW AND auto_quarantine is enabled → Plan Library: transition plan to QUARANTINED → Record lifecycle action in training run report → Emit alert (via hooks)6.3 Thresholds
Section titled “6.3 Thresholds”{ training: { promotion_threshold: 0.95, // 95% pass rate to promote quarantine_threshold: 0.50, // Below 50% pass rate to quarantine }}6.4 Training Results Feed Trust Score
Section titled “6.4 Training Results Feed Trust Score”After each training run, the plan’s trust_signals.training_pass_rate is updated. This feeds into the trust score calculation (see Plan Library Specification, Section 5).
7. Dataset Authoring
Section titled “7. Dataset Authoring”7.1 File Format
Section titled “7.1 File Format”Datasets are JSON files:
{ "dataset_id": "ds_send_welcome_email", "name": "Send Welcome Email - Core Scenarios", "version": "1.0.0", "target_plan_id": "plan_send_welcome_email", "execution_mode": "TRAINING", "environment": "development", "cases": [ { "case_id": "happy_path", "name": "New customer receives welcome email", "intent": { "mode": "plan_direct", "plan_id": "plan_send_welcome_email", "version": "2.0.0", "inputs": { "customer_id": "cust_001" } }, "fixtures": { "seed_data": { "customers": [ { "id": "cust_001", "name": "Jane Doe", "status": "active" } ] }, "mock_responses": { "send_email": { "message_id": "msg_test_001", "status": "sent" } } }, "expected": { "outcome": "COMPLETED", "output": { "strategy": "subset", "value": { "email_sent": true, "message_id": "msg_test_001" } }, "plan_expectations": { "must_use_capabilities": ["get_customer", "render_template", "send_email"] }, "timing": { "max_duration_ms": 5000 } } }, { "case_id": "customer_not_found", "name": "Non-existent customer returns error", "intent": { "mode": "plan_direct", "plan_id": "plan_send_welcome_email", "version": "2.0.0", "inputs": { "customer_id": "cust_999" } }, "fixtures": { "seed_data": { "customers": [] } }, "expected": { "outcome": "PERMANENTLY_FAILED", "expected_error": { "code": "CUSTOMER_NOT_FOUND", "category": "BUSINESS_ERROR", "step_index": 0 } } }, { "case_id": "planner_intent_test", "name": "Natural language intent resolves to correct plan", "intent": { "mode": "expressed", "text": "send a welcome email to customer Jane Doe" }, "fixtures": { "seed_data": { "customers": [ { "id": "cust_001", "name": "Jane Doe", "status": "active" } ] } }, "expected": { "outcome": "COMPLETED", "plan_expectations": { "must_use_capabilities": ["get_customer", "send_email"] } }, "tags": ["planner", "intent_resolution"] } ]}7.2 Dataset Location
Section titled “7.2 Dataset Location”Datasets live alongside the plans they test:
my-crm-app/ zakardo/ plans/ send_welcome_email.plan.json training/ send_welcome_email.dataset.json create_customer.dataset.json capabilities/ ...Or within a Capability Pack / Plan Pack:
@acme/crm-pack/ capabilities/ ... plans/ ... training/ ... pack.json7.3 Dataset from Traces (Auto-Generation)
Section titled “7.3 Dataset from Traces (Auto-Generation)”Developers can generate a training dataset from an existing successful trace:
$ zakardo training generate-dataset --trace <trace_id> --output ./training/my_dataset.jsonThis creates a dataset with:
- Intent from the trace’s original intent.
- Expected output from the trace’s actual output.
- Default strategy:
subsetmatch.
The developer can then refine the generated dataset (add more cases, adjust expectations, add fixtures).
8. Training Storage
Section titled “8. Training Storage”8.1 Training Runs Table
Section titled “8.1 Training Runs Table”-- zakardo_training_runs tableCREATE TABLE zakardo_training_runs ( run_id TEXT PRIMARY KEY, dataset_id TEXT NOT NULL, dataset_version TEXT NOT NULL, target_plan_id TEXT, target_plan_version TEXT, started_at TIMESTAMP NOT NULL, completed_at TIMESTAMP, duration_ms INTEGER, total_cases INTEGER NOT NULL, passed INTEGER NOT NULL DEFAULT 0, failed INTEGER NOT NULL DEFAULT 0, partial INTEGER NOT NULL DEFAULT 0, errors INTEGER NOT NULL DEFAULT 0, pass_rate REAL, average_score REAL, case_results JSONB NOT NULL DEFAULT '[]', actions_taken JSONB NOT NULL DEFAULT '[]', regression JSONB,
INDEX idx_dataset (dataset_id), INDEX idx_plan (target_plan_id), INDEX idx_started (started_at));9. Summary of Decisions
Section titled “9. Summary of Decisions”| Decision | Resolution |
|---|---|
| Plans earn trust through execution | Promotion is driven by training results + execution history, not manual assertion. |
| Training uses the real pipeline | TRAINING mode goes through Planner → Enrichment → Validator → Execution Engine. No shortcuts. |
| Three intent modes for test cases | expressed (natural language, full pipeline), structured (pre-parsed), plan_direct (skip Planner, test plan in isolation). |
| Four output comparison strategies | exact, subset, schema, custom comparator. |
| Four verdicts | PASS (1.0), PARTIAL (0.5-0.9), FAIL (0.0-0.4), ERROR (0.0). |
| Auto-promotion and auto-quarantine | Training results can automatically trigger lifecycle transitions. Configurable thresholds. |
| Regression detection | Compare current run to previous run. Flag cases that went from PASS to FAIL. Heuristic likely-cause analysis. |
| Fixtures for determinism | Seed data, mock responses, auto-gate/interaction responses ensure repeatable test cases. |
| Dataset portability | Datasets distributed in Plan Packs and Capability Packs. |
| Auto-generate datasets from traces | CLI command creates a starter dataset from a successful execution trace. |
| Auto-trigger on changes | Training runs auto-trigger on capability version change, policy change, or plan submission. |
| TRAINING mode = in-memory | No checkpointing, no outbox commits. Fixture services. Full trace at verbose level. |
10. Questions Deferred
Section titled “10. Questions Deferred”| Question | Deferred To |
|---|---|
| What is the exact CLI interface for training commands? | CLI and DX Specification |
| How does the Runtime API expose training runs? | Runtime API and SDK Specification |
| How does Zakardo Cloud aggregate training results across tenants? | PRD — Zakardo Cloud |
| How are datasets and training results displayed in the frontend? | Frontend Architecture Specification |
| How do Plan Packs and Capability Packs bundle datasets? | Registry and Marketplace Specification |