Column

The LLM API bill that looked reasonable in a demo starts looking different once real traffic hits it — and "just switch providers" doesn't fix a cost problem that's structural.

Why the per-token model breaks down at scale

Every major LLM provider — OpenAI, Anthropic, and Google — bills by tokens in and tokens out. That model is easy to reason about at the prototype stage, when a few hundred requests a day cost a few dollars. It stops being easy once a feature ships: every user session, every retry, every chunk of retrieved context for RAG multiplies the token count, and the bill scales linearly with usage you don't fully control.

The structural issue is that you're paying per unit of work on infrastructure you don't own. There's no way to amortize cost by running your own hardware more efficiently, no way to reserve capacity at a discount the way you can with cloud compute, and no visibility into the provider's margin. For a team with steady, predictable inference volume — not bursty, not occasional — that's money spent on convenience rather than on uptime or security.

Where open-source models change the calculation

Open-weight models like Llama, Mistral, and Qwen have closed most of the quality gap with proprietary APIs for well-scoped tasks: classification, summarization, extraction, and internal tooling rather than open-ended chat. When a task doesn't need the frontier model's full generality, running a smaller open-weight model yourself turns a per-token bill into a fixed infrastructure cost — a GPU or CPU-inference box you pay for whether you send it one request or a million.

Serving frameworks like vLLM and Ollama have made self-hosting far less of a research project than it was two years ago. Both expose an OpenAI-compatible API, which matters in practice: you can point an existing SDK integration at a self-hosted endpoint by changing a base URL, not by rewriting your application.

vLLM exposes that compatible endpoint with a single command:

vllm serve mistralai/Mistral-7B-Instruct-v0.3 --port 8000 --max-model-len 8192

That compatibility is also the honest caveat: self-hosting replaces a billing problem with an operations problem. Someone now owns GPU availability, model updates, autoscaling under load, and the security of an endpoint that used to be someone else's responsibility.

What self-hosting actually costs you in effort

This is the point where most teams stall. The model running on a single GPU box in a demo is not the same thing as a production inference service — it needs health checks so a stuck worker doesn't silently serve timeouts, autoscaling so a traffic spike doesn't queue every request behind a single process, and the same TLS, logging, and access control any other production API gets. None of that is specific to LLMs; it's the same checklist as any stateful service, which is exactly why running it on Kubo — standard Kubernetes with that plumbing already wired — tends to be less work than assembling it from scratch on a bare VM.

Kubernetes is also why this ends up being a Kubernetes decision more than an AI decision. Horizontal Pod Autoscaling can scale inference replicas on GPU utilization instead of CPU, readiness probes can pull an overloaded worker out of rotation automatically, and the same cluster can host the rest of your application stack next to the model. That's a meaningful part of why AI teams are standardizing their infrastructure on Kubernetes rather than running inference as a one-off service alongside a conventional cloud deployment.

Managed Kubernetes removes the part that doesn't pay off

Running your own Kubernetes control plane is a separate cost decision from running your own model, and it's worth being specific about it: the EKS/GKE/AKS bill, the on-call rotation for upgrades and node failures, and the time spent tuning autoscaling are overhead that doesn't make your inference better — it's the price of admission for using Kubernetes at all. Understanding the real infrastructure cost of running a cluster yourself usually changes the calculation: a managed platform absorbs the control-plane and node-management work, so the engineering hours a self-hosted model was supposed to save don't get spent re-learning Kubernetes operations instead.

The other decision that's easy to get wrong early is distribution size. A small team self-hosting one or two models doesn't need the same cluster footprint as a platform team running dozens of services — picking the right-sized Kubernetes distribution up front avoids paying for control-plane complexity you won't use, without giving up the autoscaling and self-healing that made Kubernetes worth it in the first place.

Latency and reliability don't take care of themselves

A hosted API absorbs a problem teams underestimate until they self-host: request queuing under load. A frontier provider spreads your traffic across a fleet you never see; a self-hosted model on a single GPU node queues requests the moment concurrency exceeds what that node can batch. The fix isn't a bigger GPU — it's the same answer as any other production service: multiple replicas behind a load balancer, autoscaled on actual utilization rather than request count, with a readiness probe that fails fast when a worker is overloaded instead of letting requests pile up behind it.

That's also where the cost math gets honest. Running three replicas of a 7B model for redundancy costs real money, and it should be weighed against the provider's bill the same way you'd weigh any other capacity-versus-cost tradeoff — not treated as a one-time migration with no ongoing overhead.

A simple way to decide

Three questions tend to cover it:

• Is your inference volume steady, not spiky? Steady volume is where fixed infrastructure cost beats per-token billing. Bursty, unpredictable traffic is where an API's elasticity is worth paying for.
• Does the task need a frontier model, or a well-scoped one? Open-weight models close the gap fast for narrow tasks. General-purpose chat with unpredictable prompts still favors the frontier APIs.
• Do you already run Kubernetes, or are you willing to? If the answer is yes, self-hosting inference is one more workload on infrastructure you're already operating — not a new platform to learn from zero.

If you answer yes to the first two and the third isn't a dealbreaker, the per-token bill is very likely the more expensive option at your actual scale, not the safer one.

Where to go from here

None of this means dropping API calls overnight — a hybrid approach, where routine or high-volume requests run on a self-hosted model and only the hardest cases fall back to a frontier API, is how most teams actually make the switch. If you're weighing that move, here's where to start before you commit to a specific stack.