Skip to content

Training & Validation

The Training System validates plans against curated test datasets, detects regressions, and automatically promotes or quarantines plans based on results.

Datasets are collections of test cases in JSON format:

const dataset: TrainingDataset = {
dataset_id: 'crm_basic_v1',
name: 'CRM Basic Scenarios',
version: '1.0.0',
cases: [
{
case_id: 'list_contacts',
input: {
intent: 'Show me all contacts',
user_context: { user_id: 'user_1', roles: ['admin'] },
},
expected: {
outcome: 'COMPLETED',
capabilities_used: ['contacts_list'],
output: {
strategy: 'subset',
value: { total: 5 },
},
},
},
],
};
StrategyDescription
exactEvery field must match exactly
subsetExpected fields must be present with matching values
schemaOutput must conform to a JSON Schema
customUser-provided comparator function
import { TrainingRunner } from '@zakardo/core';
const runner = new TrainingRunner({ timeout_ms: 30_000 });
const run = await runner.run(dataset, executePlan);
console.log(`Pass rate: ${(run.pass_rate * 100).toFixed(1)}%`);
console.log(`Passed: ${run.passed}, Failed: ${run.failed}`);

The system compares current run results against previous runs to detect:

  • New failures — Cases that previously passed but now fail
  • Score degradation — Cases with lower scores than before
  • New passes — Cases that previously failed but now pass (improvements)

The Training Lifecycle Manager automatically transitions plans:

  • Plans with pass rate >= 95% and >= 10 runs → PROMOTED
  • Plans with pass rate < 50% → QUARANTINED
  • Quarantined plans with improved scores → back to SHADOW