Some jobs do not need the most expensive model every time
Small models are often faster and cheaper for classification, extraction, and formatting. Pairing them with frontier models is usually more practical than picking a side.
Small and frontier models are not opposing camps. Stable, verifiable, high-volume tasks often fit smaller models; ambiguous or high-consequence work may need a stronger model or a human.
Route by task complexity
Test smaller models first for classification, extraction, and templated rewriting. Evaluate cross-document reasoning, complex code, and open planning separately. Route on observable task features and confidence rules.
Escalate instead of retrying forever
Escalate on validation failure, out-of-distribution input, or low confidence, then hand off to a human if the stronger model fails. Preserve idempotency keys for side effects.
Compare system outcomes
Compare end-to-end success, P95 latency, cost, escalation, and human correction, not isolated accuracy. Savings are real only after escalation and rework.
Pre-release checks
- Logs explain routing decisions
- Escalation cannot duplicate effects
- Cost includes stronger-model and human fallback
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.