Capture-loop honesty: what we discard and why.

The loop, end to end.

The capture-to-distill loop has five legible stages. Each stage is a separate subcommand or HTTP route; nothing is implicit; the boundary between stages is the place where data either gets persisted or gets dropped.

• Stage 1: the agent runs in production. Your application points its provider client at https://kolm.ai/v1/capture/<provider> and passes the real upstream key in x-upstream-api-key. The proxy forwards the call, strips the key on the way out, and observes the (input, output, latency_us, model, namespace, tenant) tuple as the response streams back.
• Stage 2: the user triages the inbox. The captured pair lands in the /captures inbox. The default state is neither kept nor discarded; it is untriaged. A human or an automated triage rule marks each pair as one of the three. Untriaged pairs do not flow forward.
• Stage 3: a winner is promoted. Pairs flagged kept become candidate distillation pairs. A namespace accumulates kept pairs until it crosses the distill threshold (default 1,000). The corpus is queryable through /v1/labels/synthesize-corpus as JSONL.
• Stage 4: the bridge produces an artifact. kolm distill --namespace tickets hands the corpus to the trainer bridge, which runs a synthesis pass, fits a LoRA adapter on the agreed base, runs the eval gate, and emits a signed .kolm. If the bridge is unreachable the route returns 503 and prints an operator hint; nothing further happens server-side.
• Stage 5: recompile or shelve. If the new artifact clears the K-score gate it is published to the registry under the same task name, with the prior receipt linked as parent. If it does not clear the gate the diagnostic JSON is written and the corpus is left intact for a re-run with different parameters.

The legibility matters because every transition is a place where a different retention policy could apply. The simple version is: at every stage we keep the smallest amount of state that lets the next stage do its job, and we surface the decision to the user with a counter.

Discard rules.

Three discard rules are in force by default. Each rule is a concession to a category of harm we have seen in adjacent products and have no interest in reproducing.

Rule one: full-text bodies decay to hashes after the namespace retention window. The default window is 30 days from the capture timestamp. Inside the window the inbox shows the full prompt and the full response so the triager can read them and make a real keep-or-discard decision. After the window, the body is hashed (SHA-256 over the canonical UTF-8 text), the original is deleted from the row, and the hash is what persists. Counters and dedup still work; the text is gone.

Rule two: the namespace is the retention boundary, not the tenant. Two namespaces inside the same tenant can have different retention windows. A customer-support namespace might keep raw bodies for 7 days because the support tickets contain PII. A code-review namespace might keep raw bodies for 90 days because there is no PII risk and the long tail of code patterns is worth more. The CLI sets the window per namespace; the dashboard shows it next to the namespace name; the audit log records every change.

Rule three: no implicit-signal collection without opt-in. A naive system can guess at "user happiness" by watching which responses get retried, which get copy-pasted, or how long a session lasts. We do not do this by default. The only feedback signals the loop reads are explicit: a kept-or-discarded flag from the inbox, or a structured feedback object the user POSTs to /v1/captures/:id/feedback. If the user opts a namespace into implicit signal collection, the dashboard shows the toggle as enabled and the receipt for any downstream distill records the source as signal: implicit.

The pattern is the same in all three rules: the default is conservative; opting in is explicit; the decision is surfaced. None of this is novel. What is unusual is that the rules are visible to the deployer without reading source code.

Local loop, not closed loop.

The framing matters because "closed-loop AI" is a phrase that gets used loosely. A closed loop in control theory observes its own output, compares to a setpoint, and adjusts continuously. The capture-to-distill loop is not that. It is a local loop with explicit handoffs.

The handoff that makes it local is the bridge boundary. The kolm server holds the corpus and the triage state. The trainer bridge holds the GPU and runs the actual distillation. The artifact, once produced, is signed and delivered back to the customer's storage; the server does not retain a copy of the bytes. The trainer bridge runs as a separate process the customer can host themselves, run on a partner GPU, or call out to a managed instance; whichever option is chosen, the bridge is the place where training actually happens, and the server is the place where the metadata lives.

This design has three concrete consequences.

• The compile is human-initiated. A namespace reaching 1,000 kept pairs does not automatically trigger a distill. It unlocks the distill command. A human runs kolm distill --namespace tickets; the trainer bridge accepts or refuses; the artifact is signed; the user inspects the diff against the previous version. No automatic deployment.
• The adapter is evaluated before it ships. The distill pass terminates in the K-score gate. An adapter that improved 12 cases and regressed 9 will lose net K-score and the compile will fail; the corpus stays put; the operator gets a diagnostic. There is no "shadow deploy" of an unevaluated adapter.
• The chain is auditable. The receipt for the new artifact carries a parent field pointing to the prior version, plus the corpus hash, the bridge identity, and the K-score delta. A third party reading the chain can reconstruct every distill decision in order.

What goes wrong if you keep everything.

The temptation to retain every body for the longest possible window is real. More data is more option value; more option value is more leverage during the next compile. We resist it because four kinds of harm compound when the retention horizon is unbounded.

PII bleed. A support inbox accumulates names, addresses, account numbers, and the occasional unredacted credit card. Three years of those tuples are a liability shaped exactly like a breach. The 30-day hash-down rule means the worst breach against a stale capture row produces hashes, not bodies. Hashes are not zero risk, but they are a substantially smaller risk surface than full plaintext.

Distribution drift. An agent trained on a body that is 18 months old is trained on a world that has moved. Catalogs changed; case law updated; pricing rotated. A corpus that retains all bodies forever produces an adapter that is anchored to a past that no longer exists. Aggressive decay forces the loop to keep up.

Privilege violations. A namespace inherits the access control of the tenant that captured it. A multi-team deployment where retention is unbounded eventually accumulates bodies that one team produced and another team should not see. Tight retention windows compress the privilege-violation horizon to something a routine audit can cover.

Regulatory exposure. HIPAA, GDPR, and the ICO breach-notification regime all treat "data we kept past its useful life" as worse than "data we did not collect". A short retention window is a defensible position; an unbounded retention window is an unbounded promise.

What you give up by discarding.

The honest cost of conservative defaults is real. Three categories of evidence get lost in the discard window.

Long-tail edge cases. A bug only fires on the third Tuesday of months with an R in their name. The kept-pair from that incident is 31 days old when the next reproducer would help; the body has decayed to a hash. The operator either re-runs production until the case re-occurs, or they bump the retention window for that namespace before the next reproducer is needed.

Rare failure patterns. A class of inputs that causes a refusal cascade. The captures exist, but only as scattered rows across three or four namespaces. The aggregation that would tell you the cascade has a shared root is gone after 30 days because each row is now a hash. The mitigation is a structured failure tag at capture time; the cost is that the operator has to think about the failure shape before the data ages out.

Post-deployment validation. A theory you want to test about how an artifact behaves on January's traffic is not testable in March if January's bodies are gone. The mitigation is structured holdout: a small fraction of every namespace's traffic is sampled with full bodies retained for the longer window, the rest decays. This is the per-namespace retention setting expressed concretely.

You can have conservative defaults or you can have a complete archive. You cannot have both. The default is conservative; the opt-in is per-namespace and audited; the choice belongs to the deployer.

The four counters on the dashboard.

The capture dashboard surfaces four counters per namespace. The counters are the operational vocabulary of the loop. Every action and every dropped row updates one of them.

Counter	What it counts	Source field
captured	Total tuples observed since namespace creation. Monotonic.	captures.count
discarded	Tuples flagged discarded in the inbox, or aged out by the retention window. Their hash persists; the body does not.	captures.discarded
distilled	Kept pairs that have been included in a successful distill run.	captures.distilled
merged	Distill runs that cleared the K-score gate and were published to the registry under this task name.	captures.merged

The relationship between the four is the diagnostic. A namespace where captured is 12,000 and discarded is 11,800 has a noisy upstream; the triage rule needs to be tightened. A namespace where distilled is 4,000 and merged is zero has a gate problem; the corpus is being submitted but no run is clearing it. The CLI prints the four numbers on every kolm capture status --namespace <n>; the inbox shows them at the top of the page; the receipt for any merged artifact records all four at the time of the merge.

Opt-in retention per namespace.

The defaults are conservative because the population of customers is heterogeneous and the failure mode of an aggressive default is irreversible. A customer with a stronger compliance posture, or a customer running on-premise where the data does not leave their network, may rationally want more retention. The opt-in is per-namespace, the configuration is auditable, and the dashboard shows the current state at all times.

$ kolm capture status --namespace tickets
captured:    14,203
discarded:    3,118  # 22% triage discard rate
distilled:    8,400  # included in 6 distill runs
merged:           4  # 4 of 6 cleared the gate

retention:    30d    # default
implicit:     off    # default; only explicit kept/discard flags read
signal:    {kept,feedback}
ready:       yes     # run: kolm distill --namespace tickets

The customer reads four numbers and three switches. They know what got captured, what got dropped, what got used, and what got shipped. They know the window. They know whether implicit signal is on or off. They know whether a distill is unlocked. None of this requires reading source code; none of it requires trusting a vendor narrative.

That is the whole pitch on retention. The defaults are conservative because the defaults have to be defensible without case-by-case review. The opt-ins exist because some workloads earn more retention. The counters exist because a system that drops your data without telling you is indistinguishable from a system that lost it. One inbox, four counters, three switches, a 30-day default decay. The rest is the deployer's call.