An open model has no API bill. It is not necessarily cheap.
GPUs, operations, inference tuning, and downtime all land on the same ledger. Self-hosting only makes sense after you have done the math on volume and people.
Open weights remove per-token payment to a model vendor, not compute, engineering, monitoring, or risk costs. Commercial APIs also include rate limits, data terms, and vendor dependency beyond list price.
Calculate a year of workload
For self-hosting include GPU capacity, idle time, storage, bandwidth, inference serving, upgrades, on-call work, and security. For APIs include tokens, caching, retries, egress, and integration maintenance.
Verify constraints before self-hosting
Residency, offline operation, custom weights, or steady high throughput may justify hosting. Bursty low volume and small teams usually favor APIs. “Data stays here” does not replace internal access controls.
Find break-even with a pilot
Benchmark throughput, P95 latency, quality, and recovery on target hardware and real context lengths. Calculate cost per successful task at several utilization levels; comparisons without utilization assumptions are misleading.
Pre-release checks
- Labor and idle capacity are priced
- Quality uses the same set as API candidates
- Compliance requirements are documented
Sources
Related
Writing, extraction, long reasoning, and high-volume support do not need the same model. Cost, latency, privacy, and reliability rarely point to one name.
Small models are often faster and cheaper for classification, extraction, and formatting. Pairing them with frontier models is usually more practical than picking a side.