What this recipe does
NPS post-survey free-text is a goldmine that nobody reads. This recipe routes every response into your existing theme taxonomy (themes.json: pricing, performance, onboarding, integrations, support_quality, feature_gaps, mobile, billing). The verifier enforces that every output theme is one of the taxonomy entries — no novel themes invented at run time.
The intent_to_churn and mention_competitor fields catch the two signals you most want to alert on, separately from the score.
The spec
{
"output_kind": "json",
"schema": {
"required": ["score_band", "primary_theme", "intent_to_churn", "mention_competitor"],
"properties": {
"score_band": { "enum": ["promoter", "passive", "detractor"] },
"primary_theme": { "$ref": "themes.json" },
"secondary_themes": { "type": "array", "items": {"$ref": "themes.json"} },
"intent_to_churn": { "type": "boolean" },
"mention_competitor": { "type": ["string", "null"] },
"quote": { "type": "string", "description": "verbatim phrase to surface" }
}
},
"verifier": {
"theme_must_be_in_taxonomy": true,
"quote_must_appear_in_input": true,
"competitor_list": "competitors.txt"
}
}
Gold pairs (3 of 500 shown)
input
score: 9 text: love how fast the app loads now, big improvement over last quarter. mobile is still rough though.
output
{
"score_band": "promoter",
"primary_theme": "performance",
"secondary_themes": ["mobile"],
"intent_to_churn": false,
"mention_competitor": null,
"quote": "love how fast the app loads now"
}
input
score: 3 text: support takes 4 days to respond and we're already evaluating Pinwheel which has dedicated CSMs for our tier. Likely switching at renewal unless something changes.
output
{
"score_band": "detractor",
"primary_theme": "support_quality",
"secondary_themes": ["pricing"],
"intent_to_churn": true,
"mention_competitor": "Pinwheel",
"quote": "likely switching at renewal unless something changes"
}
input
score: 7 text: it works fine. nothing wrong, nothing amazing.
output
{
"score_band": "passive",
"primary_theme": "feature_gaps",
"secondary_themes": [],
"intent_to_churn": false,
"mention_competitor": null,
"quote": "nothing wrong, nothing amazing"
}
Compile
kolm compile "NPS classifier with closed-vocab themes" \ --base qwen2.5-coder-3b \ --pairs pairs.jsonl \ --taxonomy themes.json \ --competitors competitors.txt \ --verifier closed-vocab \ --k-floor 0.85 \ --output nps-classifier.kolm ok wrote nps-classifier.kolm k_score=0.90 signature=hmac-sha256
K-score gate
Run-time profile
M2 MacBook
580ms
RTX 5090
160ms
iPhone 15 Pro
1.7s
CPU x86 (server)
2.2s
Deploy
# delighted webhook receiver — auto-tag every response, alert on churn intent: on_response = (r) => { const tag = kolm.run('nps-classifier.kolm', r); airtable.update(r.id, tag); if (tag.intent_to_churn) slack.post('#csm-alerts', tag); if (tag.mention_competitor) slack.post('#competitive-intel', tag); };