Don't put a model where you need a reproducible answer

I build with language models daily. I have also, three times in the last year, pulled one out of a system and replaced it with a lookup table and some regular expressions — and each time the system got better, not worse. The interesting part is that the three problems all looked like model problems. Classification, judgement, natural language in and a category out. If you sketched any of them on a whiteboard, the obvious implementation is a prompt. ## The two questions Before reaching for a model, I now ask two things. They are not about difficulty, and they are not about cost. **Does the same input need to produce the same answer next Tuesday?** **Do I need to explain the answer to someone who disagrees with it?** If either is yes, a model is the wrong tool — not because it will be inaccurate, but because it will be *unaccountable*. And accountability, not accuracy, is usually what these systems actually need. ## One: a compliance classifier I built a tool that classifies growth and marketing tactics as green, amber or red against the rules that actually govern them — CAN-SPAM, CASL, TCPA, GDPR. It runs offline, in Python, with no model call anywhere in the classification path. The register behind it is 68 tactics across 9 channels, with 105 tests. The version with an LLM was tempting and I did not build it, for one reason: a model asked whether ringless voicemail is legal will give you a fluent, confident paragraph, and it will occasionally cite a statute that does not say what it claims. When the output is "yes, you may send this", a hallucinated citation is not a quality problem. It is the thing that ends up in front of a regulator. So the classifier is a lookup with conditions attached, and every verdict carries the citation that produced it. If you disagree with a verdict you can open the register, read the rule, and argue with a human decision that somebody wrote down. You cannot argue with a temperature. **Unknown resolves to amber, never green.** That is the other half. A deterministic system can be honest about its own coverage; a model will always produce an answer, and the confident wrong answer is the expensive one. ## Two: an industry classifier My CRM files every business into one of 14 categories, from inputs that are reliably messy — Google's category taxonomy calls title companies "insurance agencies" because they sell title insurance, and calls bookkeepers "Services". Textbook LLM task. I wrote regex rules instead, with 26 tests. The reason is what the field is *for*. The category decides which case study a prospect gets sent and which page they land on. A misfiled record is invisible — it sits in the wrong filter, quietly, and eventually somebody gets the wrong email. There is no error, no exception, no alert. The only defence is that the rules are inspectable and the failures are reproducible. And they were. My first version wrapped every token as `\b(chiropract)\b`, which cannot match "Chiropractor" — the trailing word boundary fails on a prefix. Every chiropractor in the database landed in "Other", along with every plumber, because `plumb` cannot match "Plumber" either. That bug is the argument, not a counterexample to it. It was found by a test, fixed in one line, and pinned by a case so it cannot come back. A prompt that misclassifies chiropractors gives you nothing to fix and no way to know it happened. ## Three: a copy linter The third is the one that changed how I think about all of this. I let a model write marketing copy for my own website. It produced four statistics: a project count, a satisfaction percentage, a claim about years of experience, and a claim about how many businesses I had worked with. All four were plausible. All four read well. None of them had happened, and I did not catch a single one by reading — I found them weeks later while auditing something else. The fix was not a better prompt. "Do not invent statistics" is a wish, and wishes do not survive contact with a system that generates plausible text for a living. The fix was a register: every figure the copy is permitted to contain, each with a source, and a deterministic checker that rejects any number outside it. It holds 21 facts, three of them marked internal and unpublishable. This post went through it: every figure above — the 68 tactics, the 105 tests, the 14 categories — is traceable to an entry with a source, or it would not be in the sentence. The checker has no model call in it and never will, and that constraint is written at the top of the file. A model asked "is this copy honest?" will approve a figure it invented moments earlier, because it has no privileged access to whether the figure is true. It is the same generator, asked a different question. ## So when is a model the right answer? I am not arguing against them. I ship them. Use one where the output is **read by a human who will judge it** — drafting, summarising, explaining, translating, generating options. Where being slightly different each time costs nothing, and where the person receiving it has both the context and the authority to reject it. The pattern in all three of my systems is the same, and it is not "no models". It is: **the model drafts, and something deterministic decides.** The copy linter runs on model-written copy. The classifier could feed a model that writes the outreach. The generator and the gate are different components, and the gate is the one you can audit. ## The bit that actually matters Reproducibility is not a nice-to-have you trade away for capability. It is what makes a system *reviewable*, and a system nobody can review is one nobody can correct. If you cannot re-derive last Tuesday's answer, you cannot investigate a complaint about it. If you cannot point at the rule, you cannot argue about the rule — and if you cannot argue about the rule, it never improves. That is worth more than the accuracy you give up. Usually you do not give up any. --- The multi-tenant platform behind some of this — 95 TypeScript source files, 29 test files, 36 migrations, built and owned by me with no live client — is public as an architecture reference: **[github.com/jaklabs/telehealth-platform-reference](https://github.com/jaklabs/telehealth-platform-reference)** Before taking my word for anything, the **[free website check](/website-audit)** is a public unauthenticated endpoint that drives headless Chromium at an arbitrary URL from a stranger — an SSRF liability unless the boundary is real. It is a fair sample of how I build the parts nobody looks at. If you have shipped an AI feature and cannot answer how you would know if it were wrong, that is the work I do. [Tell me what you built](/contact).