Every .kolm carries its own evals.

What the harness measures

When you compile a kolm artifact, the eval set you gated it on is sealed inside the bundle as evals.json. kolm bench reads that, runs every recipe in recipes.json against every embedded case, and reports five things:

Every cell above maps to a line in src/benchmark.js. If you don't trust us, run the harness yourself.

The compile-time gate

Before an artifact is signed and sealed, it has to pass a sandbox verifier (src/verifier.js). The verifier runs the candidate generator against three sets (positives, negatives, property tests) and computes a quality score:

If the gate fails, the artifact is never written to disk. There is no "bad kolm" that ships and then fails in production. The build either produces a binary that passed its own evals, or it produces nothing.

The verified-inference math

For artifacts whose ground-truth labels come from a frontier model, kolm uses k-sample verified inference (src/verified.js). Given a stochastic generator with single-shot accuracy p on a verifiable task, and a sound verifier with single-shot accuracy v, drawing k independent samples and accepting the first that the verifier passes yields:

This is the SOTA-amplifier. It is not a claim; it is monotone in k by construction, and the verifier's soundness is what bounds it. Worked numbers for a verifiable coding task with p≈0.91 and v≈1.0:

The amplifier only fires when the verifier is sound (no false positives). For non-verifiable tasks (open-ended generation, judgment calls), v ≪ 1 and the amplifier collapses. We are explicit about which tasks fall in each bucket.

K-score, computed

K-score is what the manifest reports: a single number that combines accuracy, eval coverage, and binary size. Bigger is better; bloat is penalized:

A 5KB recipe at 100% accuracy and 100% coverage scores ~588. A 50MB checkpoint at the same accuracy scores ~62. The denominator is the discipline.

→ how K-score is computed, all five raw axes

Reproducer

Every kolm artifact you compile, you can bench against on your own machine. The harness is one file.

# compile a sample artifact
kolm compile "classify support emails" \
  --examples examples.jsonl \
  --out support.kolm

# run the embedded eval set offline
kolm bench support.kolm --runs 10 --out report.json

# report.json conforms to spec kolm-benchmark-1 (real run against test/fixtures/sample.kolm)
{
  "spec": "kolm-benchmark-1",
  "artifact_sha256": "sha256:b8344082…1089880e0",
  "artifact_bytes": 3259,
  "k_score": 424.57,
  "evals": { "n": 4, "graded": 400, "passed": 400, "accuracy": 1.0 },
  "latency_us": { "p50": 274, "p95": 335, "max": 639 },
  "privacy": { "runtime_egress_attempts": 0, "blocked": false },
  "integrity": { "signature_valid": true, "receipt_chain_steps": 5 }
}

The harness patches fetch, http, https, net, tls, and dns before any recipe runs. Any egress attempt is recorded and counted against the artifact. This is how you prove a binary is sovereign.

Four real fixtures, four real runs

Every number below comes from running kolm bench against the four signed fixtures shipped in this repo, on Windows 11 + Node v24.14.0. Anyone can rerun the exact commands. The fixtures are the same artifacts the cookbook and /build-your-own templates produce. kolm-benchmark-1 is the report schema.

$ RECIPE_RECEIPT_SECRET=kolm-public-fixture-v0-1-0 \
  kolm bench test/fixtures/<name>.kolm --runs 50 \
  --target kolm-public-v0.1.0 --device win11-node24

Each row is a 50-run benchmark. The fixtures sign under RECIPE_RECEIPT_SECRET=kolm-public-fixture-v0-1-0; this is also the secret the smoke battery and the e2e tests use. Each artifact ships its own embedded eval set, so the harness re-runs those evals offline with fetch/http/https/net/tls/dns all monitored. Latency varies with hardware; evals.accuracy, privacy.runtime_egress_attempts, and integrity.signature_valid do not. Reproducer notes at docs/benchmark-results-v0.1.0.md.

What we do not claim

kolm does not appear on the SWE-bench Verified leaderboard. It does not claim to make Opus or any frontier model better at coding in general. The compile-time gate either produces an artifact that passes its embedded evals, or it doesn't ship.

Verified inference (k-sample + sound verifier) is monotone in k by construction. The interesting empirical question is what the cost-vs-accuracy curve looks like for a given task and verifier; that is the per-task compile decision, made in your account, not on a leaderboard.

If your task isn't verifiable (no machine-checkable ground truth), kolm tells you. The amplifier doesn't fire and the artifact isn't sealed. We say no on builds we can't gate.