Slack thread summarizer recipe

What this recipe does

You return from PTO. The #project-foo channel has a 240-message thread that started while you were gone. Reading it cold takes 25 minutes. This recipe reads it for you in 2 seconds and returns the 3-block summary you actually want. The verifier enforces shape (3 sections, every "decision" cites the message id where it was made, every "open question" cites the most recent unresolved message).

The spec

{
  "output_kind": "json",
  "schema": {
    "required": ["tldr", "decisions", "open_questions"],
    "properties": {
      "tldr": { "type": "string", "maxLength": 320 },
      "decisions": { "type": "array", "items": {
        "required": ["text", "by", "msg_id"],
        "properties": {
          "text": { "type": "string" },
          "by": { "type": "string" },
          "msg_id": { "type": "string" }
        }
      } },
      "open_questions": { "type": "array", "items": {
        "required": ["question", "asked_by", "last_msg_id"],
        "properties": {
          "question": { "type": "string" },
          "asked_by": { "type": "string" },
          "last_msg_id": { "type": "string" }
        }
      } }
    }
  },
  "verifier": {
    "redact_before_train": true,
    "msg_id_must_exist_in_input": true
  }
}

Gold pairs (1 of 100 shown)

input - 38-message thread (truncated)

[1761843102.001 sara] proposed: switch the worker queue from sqs to redis-streams
[1761843223.005 luis] +1 — sqs receive-loop tax is killing us at 50qps
[1761843541.008 ana]  concern: ops surface for redis is harder than sqs
[1761844210.012 luis] benchmarks attached, redis-streams 4x cheaper at scale
[1761845882.018 sara] decision: roll redis-streams behind a feature flag, sqs stays as fallback
[1761846430.021 ana]  who owns the rollout?
[1761847001.024 sara] luis owns rollout, ana owns ops doc
[1761848112.030 ?]    open: how do we handle the 14k in-flight sqs messages at cutover?

output

{
  "tldr": "Team agreed to migrate worker queue from SQS to Redis Streams behind a feature flag (SQS as fallback). Luis owns rollout; Ana owns the ops doc. Open: how to drain in-flight SQS messages at cutover.",
  "decisions": [
    {"text": "Migrate worker queue from SQS to Redis Streams behind a feature flag; SQS as fallback.", "by": "sara", "msg_id": "1761845882.018"},
    {"text": "Luis owns rollout; Ana owns ops doc.", "by": "sara", "msg_id": "1761847001.024"}
  ],
  "open_questions": [
    {"question": "How do we handle the 14k in-flight SQS messages at cutover?", "asked_by": "thread", "last_msg_id": "1761848112.030"}
  ]
}

Compile

kolm compile "summarize slack thread into TLDR + decisions + open questions" \
  --base qwen2.5-7b-instruct \
  --pairs pairs.jsonl \
  --redactor secrets.regex \
  --verifier schema:thread-summary.json \
  --k-floor 0.80 \
  --output thread-summarizer.kolm

ok wrote thread-summarizer.kolm
   k_score=0.83  signature=hmac-sha256

K-score gate

K-score 0.83 held-out 50 threads · schema-pass 100% · msg-id-resolve 100% · user-rated useful 86%

The msg-id-resolve check kills the most common hallucination ("decided to do X" with no actual message that says X). Every cited msg_id must appear in the input verbatim or the output is rejected at compile time.

Run-time profile

M2 MacBook

2.8s

RTX 5090

630ms

iPhone 15 Pro

7.2s

CPU x86 (server)

9.1s

Deploy

# slash command — `/recap` on any thread:
on_slash_recap() {
  thread=$(slack thread-export "$1")
  summary=$(echo "$thread" | kolm run thread-summarizer.kolm --input-stdin)
  slack post-ephemeral "$summary"
}

"What did I miss?"