Support-reply recipe

What this recipe does

Drafts a first-pass reply, never the final send. Output goes into the support agent's draft window, not directly to the customer. The verifier enforces that any factual claim in the reply (a feature exists, a behavior is intended, a fix is available) is grounded in your KB corpus — outputs that promise something the KB doesn't say are dropped from training.

The spec

{
  "output_kind": "json",
  "schema": {
    "required": ["reply", "kb_articles_cited", "requires_human_review"],
    "properties": {
      "reply": { "type": "string", "maxLength": 1200 },
      "kb_articles_cited": { "type": "array", "items": {"type":"string"} },
      "requires_human_review": { "type": "boolean" },
      "sentiment": { "enum": ["calm", "frustrated", "angry", "praise"] }
    }
  },
  "verifier": {
    "kb_corpus": "kb/",
    "factual_claim_must_be_grounded": true,
    "style_corpus": "prior-replies/",
    "escalate_on_anger": true
  }
}

Gold pairs (1 of 200 shown)

input

subject: export to CSV broken
body: I tried exporting my saved list to CSV and got a 502. Tried 3 times,
same result. This used to work last week. Pretty frustrating, I have a
deadline today.

output

{
  "reply": "Thanks for the report — sorry that hit you under deadline. The CSV export endpoint had an issue this morning that we shipped a fix for at 14:22 PT. If you can retry the export now, it should work; if you still see the 502 in the next 10 minutes, reply to this thread and I'll pull it up myself. As a backup, the JSON export at /export/saved.json includes the same fields and we can convert it for you if you prefer.",
  "kb_articles_cited": ["kb/exports.md#csv", "kb/exports.md#json-fallback"],
  "requires_human_review": false,
  "sentiment": "frustrated"
}

Compile

kolm compile "draft support reply, KB-grounded, brand voice" \
  --base qwen2.5-7b-instruct \
  --pairs pairs.jsonl \
  --kb-corpus kb/ \
  --style-corpus prior-replies/ \
  --verifier ground-and-style \
  --k-floor 0.82 \
  --output support-reply.kolm

ok wrote support-reply.kolm
   k_score=0.85  signature=hmac-sha256

K-score gate

K-score 0.85 held-out 60 tickets · ground-pass 96% · style-match 91% · agent-edit-distance < 20% on 78%

"Edit-distance < 20%" is the practical metric: 78% of held-out replies needed less than 20% character-level edits before the agent sent them. The remaining 22% needed more substantive rework or were flagged for escalation.

Run-time profile

M2 MacBook

2.0s

RTX 5090

470ms

iPhone 15 Pro

5.4s

CPU x86 (server)

6.8s

Deploy

# zendesk app — drops a draft into the agent's reply pane:
client.on('ticket.opened', async (t) => {
  const draft = await kolm.run('support-reply.kolm', {
    subject: t.subject, body: t.body, customer_tier: t.requester.tier,
  });
  if (draft.requires_human_review) ZAFClient.routeTo('tier-2');
  else ZAFClient.draftReply(draft.reply);
});

Support replies in your voice.