Decision Models vs Big Models: When a Tiny Model Is the Right Tool
A two-billion-parameter model can answer with a label for less than a fraction of a cent. That changes what is worth automating.
Here is a distinction that gets lost in the AI discourse: not every question needs a model that can write an essay. A new category has appeared over the last year - decision models - and they are quietly changing what gets automated at scale.
A decision model does one thing: given some input and a typed question, it returns a structured choice. Spam or not. Route here or there. Approve or escalate. Cloud or self-hosted. The output is a label with a score, not prose. Because it only has to make one narrow call, it can be tiny - a couple of billion parameters - and fast enough to run inside a request, on a laptop, or on a phone.
The economics are the part that changes decisions. Perplexity's decider runs at roughly four cents per million input tokens with no output charge. Cloudflare's is similar. Some run locally in under ten milliseconds. Do the arithmetic on a pipeline that classifies a million items a month and the difference between a general model and a decision model is the whole project budget.
Where they fail is the part everyone discovers too late: decision models are narrow by construction. Ask one to summarize a document, write a reply, or reason across two conflicting pieces of context and you get something confidently small and wrong. They also inherit whatever bias sits in their training - a classifier trained on past decisions will reproduce past decisions, including the bad ones, without any ability to notice the pattern.
The architecture that works is layered, and this is the genuinely useful part. Put the decision model in front, where it filters the routine traffic - classify, route, score - and let only the uncertain cases escalate to the big model. Most of your volume never needs the expensive tier, and the expensive tier gets to spend its budget on the cases where a human is actually waiting.
The mental model I keep coming back to: use a model that writes prose when you need prose, and use a model that returns a label when you need a label. People reach for the frontier model by default because it is the one they know, and then quietly pay for paragraphs they were going to throw away. The question worth asking is not how smart the model is, but what shape the answer needs to be.
Frequently asked questions:
Q: What is an AI decision model?
A: A small model built to return one typed answer - a label, a score, a routing choice - in a single pass, instead of generating text. It is typically a fraction of the size and cost of a general assistant, which is what makes high-volume automation affordable.
Q: When should I use a decision model instead of ChatGPT or Claude?
A: When the task is classification in disguise - spam filtering, ticket triage, routing, approve-or-escalate, deduplication - and the answer is a label. Use a general model when you need prose, reasoning across context, or anything a person will read.
Q: Do decision models replace big models?
A: They replace big models at the front of the pipeline, not in the middle of it. The pattern that works is cheap classification first, expensive generation second, which is how you get most of the volume handled for a fraction of the cost.