Most companies should not train or serve their own AI model. Below a specific, sustained volume, an API is cheaper than running your own GPUs, even before counting the engineer needed to keep a self-hosted model online. Self-hosting only wins for a narrow profile: steady, pooled, high-volume demand with ML infrastructure talent already on staff. If that isn't your company, keep paying per token and revisit the math when volume changes.
One-sentence outcome: you'll leave with one calculation, token volume against a breakeven threshold, that tells you whether self-hosting saves money or is a trap.
Technical companion, optional: Inference Engineering: TTFT, TPOT, and Two Clocks.
What this post covers
- What inference is, and where it sits in the AI stack.
- The two paths: API vs. train and/or serve your own model.
- A cost breakdown: the formula, a $/1M-token comparison, and a breakeven volume.
- Non-cost reasons that push toward one path or the other.
- The exact ICP for self-hosting, and why most readers aren't it.
Inference is the cost you pay every time someone asks the model something
Training happens once and produces a fixed set of weights. Inference is every use afterward: prompt in, tokens out. If training is R&D, inference is cost of goods sold, paid on every unit, forever.
Inference sits after training and context assembly, right before the answer reaches the user. Two levers set its price:
- Compute time per token: GPU work per token produced.
- Idle time: paid-for compute sitting unused between requests.
One support-agent question is one inference call. A multi-turn chatbot session is dozens. Per-token price times volume is the AI line on the P&L.
Renting by the token or building the factory: two different bets
Pay per token via an API and you send text, get text back, and pay for what you used. There's no infrastructure to run and no idle-capacity risk, because the vendor absorbs it.
Train and/or fine-tune a model, then serve it yourself, typically on rented GPUs, and you own the weights, uptime, autoscaling, and failure modes instead. It's a fixed-cost factory you must keep busy to make cheap; idle, it's expensive.
There's a middle path too: managed providers serving open-weight models for you (~$0.88/1M blended, 70B-class model, Together AI), open-weight economics without owning GPUs or headcount.
Self-hosted inference is only cheap once GPU utilization holds above roughly 70%
Formula: $/1M output tokens = (GPU $/hr ÷ 3,600 sec) ÷ tokens/sec × 1,000,000, ÷ utilization rate.
An H100 at $2.50/hr, ~1,200 output tokens/sec (realistic production throughput, not a synthetic peak benchmark), works out to roughly $0.64/1M tokens at 90% utilization and $1.16/1M tokens at 50% utilization. Cost nearly doubles, since idle GPU-hours are paid regardless of traffic. Utilization, not GPU speed, decides self-hosted economics: a well-run fleet runs 50-70%, and 90%+ needs high, steady, pooled demand.
Against representative API pricing (Aug 2026, public rates): cheap tier ~$0.10-$1.00/1M blended, mid tier ~$8-$15/1M output, flagship ~$25-$30/1M output. Self-hosted at 90% ($0.64) lands inside the cheap tier's range, so it doesn't clearly beat the cheapest APIs on GPU cost alone. At 50% ($1.16) it already exceeds the cheap tier's top end. The advantage only shows up against mid and flagship pricing.
Layer in the fixed cost: self-hosted inference needs at least one MLOps engineer, $150K-$300K/year, largely fixed regardless of volume. Breakeven volume V (millions of tokens/year): V = headcount ÷ (API price − self-hosted GPU cost).
- Vs. mid tier (~$8/1M) at 90% utilization: V ≈ $150K ÷ $7.36 ≈ 20.4B tokens/year, rising to ≈ 40.8B at $300K headcount. That's roughly 1.7-3.4B tokens/month, sustained.
- Vs. the cheap tier (~$1.00/1M): the gap shrinks to $0.36, pushing breakeven to roughly 35-70B tokens/month, reached only by a handful of AI-native platforms.
Volume must also be steady: spiky traffic drags utilization down and erases the advantage. Below these thresholds, cheap-tier APIs are hard to beat once headcount and idle GPU-hours are priced in honestly.
Fine-tuning itself is cheap: a QLoRA fine-tune of a 70B model runs ~$15-30 in GPU-hours, and a full fine-tune runs ~$200-300. Headcount and idle GPU-hours, not customization, are what make self-hosting expensive.
These figures come from public pricing and cost data, not internal estimates: OpenAI and Google list API pricing on their official developer sites; Together AI publishes managed open-weight pricing; GPU rental rates are cross-checked across multiple providers; MLOps salary data comes from salary aggregators; fine-tuning cost ranges come from published GPU-hour budget guides. Vendor pricing moves often, so treat these as directional, not fixed.
Cost isn't the only variable: four more reasons to self-host anyway
- Data residency and compliance. A healthcare or defense contractor whose data can't leave a private environment needs a controlled VPC, regardless of cost.
- Latency and control. Trading, live bidding, or safety-critical control loops, where every millisecond matters, favor a colocated model.
- Existing ML talent. A team already running GPU and Kubernetes infrastructure has a lower marginal cost to add serving, since the headcount line is already sunk.
- Deep, continuous customization. A product that needs ongoing fine-tuning on proprietary data at scale, beyond prompting or retrieval, needs control an API can't offer.
Self-hosting is the exception: here is the exact profile where it is the right call
Most companies reading this post belong in the API column. Self-hosting is the exception, not the default. It's a deliberate decision made against the volume and headcount math above, not an engineering preference.
TLDR
- Inference is the recurring cost of running a model, paid every request, unlike training, which is a one-time cost.
- Cheap-tier APIs (~$0.10-$1/1M tokens) are hard to beat once idle GPU-hours and a $150K-$300K/year MLOps engineer are honestly priced in.
- Self-hosted GPU cost only drops below roughly $1/1M tokens above ~70% utilization, and most fleets run lower than that.
- The illustrative breakeven volume is roughly 1.7-3.4 billion tokens/month sustained against mid-tier API pricing, and far higher against cheap-tier pricing.
- Self-host only if you clear both bars: sustained, steady, high-enough volume, and in-house ML infrastructure talent already on staff. Otherwise, rent by the token.
Whichever path you pick, your context layer is a separate decision
Inherent does not compete on inference: no model hosting, no GPUs, no token pricing. It sits one layer above whichever inference path a company chooses, with managed ingestion that keeps source documents current, deterministic retrieval that returns the same context for the same query, and audit logs that show exactly what a model was shown before it answered.
That decision does not change based on whether you call an API or run your own GPUs. Start with the Inherent Public API to see how the context layer works independent of your inference choice. Questions on where your company sits on the breakeven math above? DM Flow on X.
