Training & Validation
The Training System validates plans against curated test datasets, detects regressions, and automatically promotes or quarantines plans based on results.
Training Datasets
Section titled “Training Datasets”Datasets are collections of test cases in JSON format:
const dataset: TrainingDataset = { dataset_id: 'crm_basic_v1', name: 'CRM Basic Scenarios', version: '1.0.0', cases: [ { case_id: 'list_contacts', input: { intent: 'Show me all contacts', user_context: { user_id: 'user_1', roles: ['admin'] }, }, expected: { outcome: 'COMPLETED', capabilities_used: ['contacts_list'], output: { strategy: 'subset', value: { total: 5 }, }, }, }, ],};Output Comparison Strategies
Section titled “Output Comparison Strategies”| Strategy | Description |
|---|---|
| exact | Every field must match exactly |
| subset | Expected fields must be present with matching values |
| schema | Output must conform to a JSON Schema |
| custom | User-provided comparator function |
Training Runner
Section titled “Training Runner”import { TrainingRunner } from '@zakardo/core';
const runner = new TrainingRunner({ timeout_ms: 30_000 });const run = await runner.run(dataset, executePlan);
console.log(`Pass rate: ${(run.pass_rate * 100).toFixed(1)}%`);console.log(`Passed: ${run.passed}, Failed: ${run.failed}`);Regression Detection
Section titled “Regression Detection”The system compares current run results against previous runs to detect:
- New failures — Cases that previously passed but now fail
- Score degradation — Cases with lower scores than before
- New passes — Cases that previously failed but now pass (improvements)
Lifecycle Management
Section titled “Lifecycle Management”The Training Lifecycle Manager automatically transitions plans:
- Plans with pass rate >= 95% and >= 10 runs → PROMOTED
- Plans with pass rate < 50% → QUARANTINED
- Quarantined plans with improved scores → back to SHADOW