Most developers I talk to think about self-hosting LLMs vs API cost as a simple per-token comparison. “Llama 3 is free, Claude costs $3 per million tokens, done.” That math is wrong in almost every real scenario, and it’s costing people either money or performance — sometimes both.
The actual calculus involves GPU amortization schedules, electricity, engineering hours, cold-start latency, model quality deltas, and a traffic breakeven point that moves depending on your workload shape. I’ve worked through this for a production document-processing pipeline, and the numbers surprised me. Let me give you the full breakdown.
The Real Cost Components of Self-Hosting
Before comparing anything, you need an honest accounting of what self-hosting actually costs. People consistently undercount three things.
GPU Amortization
An NVIDIA A100 80GB SXM runs about $10,000–$12,000 new (2024 pricing). For most 7B–13B models, one A100 is enough. For 70B models like Llama 3 70B at full precision, you need at least 2x A100s or 2x H100s. H100s are $25,000–$30,000 each.
A reasonable amortization period for inference hardware is 3 years. That gives you a monthly hardware cost of:
- Single A100 (7B–13B models): ~$333/month
- 2× A100 (Llama 3 70B, fp16): ~$666/month
- 2× H100 (Qwen 72B, Mistral Large): ~$1,666/month
If you’re renting GPU cloud instances instead of owning hardware, those numbers look different. A100 instances on Lambda Labs or RunPod run $1.10–$2.50/hour, which is $800–$1,800/month for continuous use. Owned hardware wins quickly at high utilization; rented hardware is better for variable workloads.
Electricity
An A100 draws ~250–400W under inference load. At US commercial electricity rates (~$0.12/kWh), that’s roughly $26–$42/month per GPU for continuous 24/7 operation. Not huge, but not nothing — and at data center rates with cooling overhead, double it.
Ops Overhead: The Number Nobody Puts in the Spreadsheet
This is where the real cost hides. Running inference infrastructure means: model updates, driver updates, monitoring, restarts, debugging OOM crashes, managing quantization configs, and maintaining serving infrastructure (vLLM, Ollama, TGI). For a solo founder, that’s 4–8 hours/month minimum. For a team, it’s a fractional DevOps headcount.
At $100/hour contractor rate, 6 hours/month = $600/month. At 0.25× a $120k engineer = $2,500/month. This single line item makes self-hosting uneconomical for most teams under ~20M tokens/day.
Per-Token Cost Comparison at Current Pricing
Here’s a concrete model-by-model breakdown. API prices are current as of mid-2025.
Claude API (Anthropic)
- Claude Haiku 3.5: $0.80/M input, $4.00/M output
- Claude Sonnet 3.7: $3.00/M input, $15.00/M output
- Claude Opus 4: $15.00/M input, $75.00/M output
For most agentic workflows, Haiku handles triage and routing at ~$0.002 per typical 2,500-token call. Sonnet handles reasoning-heavy tasks at ~$0.018 per call. These numbers matter for the breakeven calculation below.
Open-Source Models: Self-Hosted Cost Per Token
Self-hosted cost per token depends on your throughput. The more tokens you push through a fixed infrastructure cost, the cheaper per-token becomes. Here’s the math for a single A100 setup running Llama 3 8B with vLLM:
- Throughput at batch size 8: ~1,500 tokens/second
- Monthly tokens at 50% utilization: 1,500 × 0.5 × 86,400 × 30 ≈ 1.94 billion tokens
- Monthly cost (A100 owned + electricity + 6hr ops): ~$1,000
- Effective cost: ~$0.52/M tokens total (input + output combined)
Against Claude Haiku at $0.80/M input + $4.00/M output, that’s genuinely cheaper — but only if you’re hitting that utilization. At 10% utilization, your effective self-hosted cost is $5.16/M tokens, more expensive than Haiku.
The Traffic Breakeven Analysis
Let’s set up the breakeven properly. Assume you’re comparing Llama 3 8B (self-hosted on one A100) against Claude Haiku 3.5 (API). Assume a typical call is 800 input tokens + 400 output tokens = 1,200 tokens.
# Breakeven calculator: self-hosting vs Claude Haiku 3.5
# Adjust these variables for your setup
MONTHLY_INFRA_COST = 1000 # USD: hardware amortization + electricity + ops
# Claude Haiku 3.5 pricing (per million tokens)
HAIKU_INPUT_PER_M = 0.80
HAIKU_OUTPUT_PER_M = 4.00
# Typical call shape
INPUT_TOKENS = 800
OUTPUT_TOKENS = 400
# Cost per API call
haiku_cost_per_call = (
(INPUT_TOKENS / 1_000_000) * HAIKU_INPUT_PER_M +
(OUTPUT_TOKENS / 1_000_000) * HAIKU_OUTPUT_PER_M
)
# Breakeven: how many calls/month to recover infra cost
breakeven_calls = MONTHLY_INFRA_COST / haiku_cost_per_call
breakeven_tokens_per_day = (breakeven_calls * (INPUT_TOKENS + OUTPUT_TOKENS)) / 30
print(f"Cost per Haiku call: ${haiku_cost_per_call:.5f}")
print(f"Breakeven calls/month: {breakeven_calls:,.0f}")
print(f"Breakeven tokens/day: {breakeven_tokens_per_day:,.0f}")
# Output:
# Cost per Haiku call: $0.00224
# Breakeven calls/month: 446,429
# Breakeven tokens/day: 17,857,143
The breakeven point for Llama 3 8B vs Claude Haiku is roughly 446K calls/month (~15M tokens/day). Below that, Haiku is actually cheaper once you account for ops. Above it, self-hosting wins on unit economics — but you’re still trading quality.
For Llama 3 70B vs Claude Sonnet 3.7, the breakeven is lower because Sonnet costs significantly more per token. With 2× A100 infrastructure at ~$1,666/month, you break even at about 92K calls/month at Sonnet pricing — that’s only ~3M tokens/day.
Quality Deltas: What the Token Math Ignores
This is where people make expensive mistakes. Running a self-hosted 8B model for a task that needs Sonnet-level reasoning doesn’t save money — it shifts cost to the humans cleaning up wrong outputs, or customers churning because the product is unreliable.
My honest quality rankings for common production tasks:
Instruction Following and JSON Output
Claude Sonnet > Mistral 7B Instruct ≈ Llama 3 8B > Qwen 2.5 7B for strict JSON schema adherence. If you’re running structured extraction at scale, Claude’s reliability advantage translates directly to fewer retry calls. If you’re already handling retries anyway, check out structured JSON output techniques for Claude — the patterns apply to both API and self-hosted scenarios.
Reasoning and Multi-Step Tasks
Claude Sonnet 3.7 and Opus 4 are genuinely ahead of open-source models at complex reasoning. Llama 3 70B and Qwen 72B are competitive for well-scoped single-step tasks. For agentic orchestration, the quality delta matters more because errors compound across tool calls.
Throughput and Latency
Self-hosted models on dedicated hardware have predictable latency. API calls have variable tail latency depending on Anthropic’s load. In practice: self-hosted vLLM on A100 gets 80–150ms TTFT for 8B models; Claude Haiku API averages 200–400ms TTFT. For real-time UX, self-hosted wins if you have the throughput to keep the GPU warm.
Misconception 1: “Open-Source Is Free”
The model weights are free. The infrastructure is not. The engineering time to set up vLLM, manage quantization (GPTQ, AWQ, GGUF), tune batch sizes for your hardware, monitor for GPU memory leaks, and keep the server running — none of that is free. I’ve seen teams spend 3 weeks getting a Llama 3 deployment stable, then discover their P99 latency is 8 seconds under load because they didn’t tune the vLLM scheduler. The API just works.
Misconception 2: “Self-Hosting Is More Private”
Often true for on-prem, but most “self-hosted” deployments actually run on cloud GPUs (Lambda, RunPod, CoreWeave, AWS). Your data still traverses the internet and lands on someone else’s hardware. The actual privacy advantage requires owning the physical compute, which puts you firmly in enterprise territory. Anthropic offers data processing agreements and doesn’t train on API data by default — the privacy gap is smaller than commonly assumed for most B2B use cases.
Misconception 3: “Quantized Models Are Almost As Good”
4-bit quantization of Llama 3 70B (roughly 35GB) lets you run it on 2× consumer A6000s or a single H100. Quality degradation is task-dependent: MMLU scores drop 1–3 points, which sounds small, but on long-chain reasoning tasks or tasks requiring precise instruction following, the errors are qualitatively worse, not just slightly worse. For batch classification or summarization, quantized 70B is fine. For complex agent reasoning, you’ll notice it.
If you’re managing costs for high-volume API usage, also look at LLM caching strategies — semantic caching can cut your effective token spend 30–50% without any model change, and the technique works whether you’re on API or self-hosted.
Model-Specific Recommendations: Llama, Mistral, Qwen
Llama 3 8B / 70B (Meta)
Best all-around open-source option. Strong instruction following, good context length (128K), permissive license for commercial use. 8B is genuinely competitive with GPT-3.5-tier for structured tasks. 70B approaches GPT-4-tier on many benchmarks. My go-to if self-hosting is justified by volume.
Mistral 7B / Mixtral 8×7B
Mistral 7B punches above its weight for code and technical tasks. Mixtral 8×7B (MoE, 47B parameters, ~13B active per token) is memory-efficient relative to quality. The sliding window attention makes it genuinely faster for long contexts. Weaker on multi-turn instruction following than Llama 3 8B.
Qwen 2.5 7B / 72B (Alibaba)
Underrated. Qwen 2.5 72B is competitive with Llama 3 70B on code and math, and the 7B model shows unusually strong multilingual performance. If you need CJK language support or strong coding ability at 7B scale, Qwen is your best option. License is permissive for most commercial uses. Deployment tooling is slightly less mature than Llama.
Deployment Infrastructure That Actually Works
For anyone who decides self-hosting is justified, here’s the stack I’d use:
# vLLM serving with continuous batching — production baseline
pip install vllm
# Run Llama 3 8B with 4-bit AWQ quantization on A100
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Meta-Llama-3-8B-Instruct \
--quantization awq \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--tensor-parallel-size 1 \
--port 8000
# This exposes an OpenAI-compatible /v1/chat/completions endpoint
# You can drop it behind nginx + auth and call it from any OpenAI SDK client
For deploying on cloud infrastructure and managing the ops side, the comparison in serverless platforms for AI agents covers how Replicate, Beam, and similar services handle model serving — relevant if you want the cost benefits of open-source models without full infrastructure ownership.
When to Use What: A Practical Decision Framework
This is the decision I’d walk you through if you asked me directly:
Use the Claude API if: you’re under 10M tokens/day, you need reliable instruction following and complex reasoning, you’re a solo founder or small team without dedicated ops capacity, or your use case is customer-facing where quality errors have real cost. The ops savings alone justify the API premium at this scale. For workflows like automated customer support, Claude’s reliability and lower retry rate matter significantly.
Self-host if: you’re consistently above 15M tokens/day, your tasks are well-defined and benchmarkable (classification, extraction, summarization), you have DevOps capacity, and you’ve run a quality eval that confirms a 7B or 70B model meets your accuracy bar. The unit economics are genuinely compelling at scale.
Hybrid approach (often the right answer): Use Llama 3 8B or Qwen 2.5 7B for high-volume, lower-stakes tasks (classification, routing, initial extraction), and route complex reasoning or high-stakes outputs to Claude Sonnet or Opus. This is what I’d implement for anything above 5M tokens/day that also needs quality on a subset of calls.
The honest summary on self-hosting LLMs vs API cost: the breakeven exists, but it’s higher than most teams expect, and it only makes sense if you’re treating infrastructure as a core competency. If you’re not, pay for the API, optimize your prompts and caching, and spend the freed engineering time on your actual product.
Frequently Asked Questions
At what token volume does self-hosting become cheaper than Claude API?
For Llama 3 8B vs Claude Haiku 3.5, the breakeven is approximately 15M tokens/day assuming a single A100 with full ops costs included. For Llama 3 70B vs Claude Sonnet 3.7, the breakeven is lower — around 3–5M tokens/day — because Sonnet’s API cost is significantly higher. These numbers shift based on GPU rental vs ownership and your actual ops overhead.
Can I run Llama 3 70B on consumer GPUs?
Yes, with 4-bit quantization (AWQ or GGUF) you can run Llama 3 70B on 2× RTX 3090s (48GB VRAM total) or a single RTX 4090 with offloading. Expect throughput around 15–25 tokens/second, which is fine for personal use but too slow for production multi-user workloads. For production, you need A100 or H100 class hardware.
How does Mistral compare to Llama 3 for production self-hosting?
Llama 3 8B generally beats Mistral 7B on instruction following and multi-turn conversations. Mixtral 8×7B (MoE) is competitive with Llama 3 70B at lower active parameter counts, making it more memory-efficient. I’d pick Llama 3 for most general-purpose tasks and Mixtral for workloads where you need 70B-tier quality but have GPU memory constraints.
Is there an OpenAI-compatible API layer for self-hosted models?
Yes — vLLM, Ollama, and Text Generation Inference (TGI) all expose an OpenAI-compatible /v1/chat/completions endpoint. This means you can swap between self-hosted models and Claude/OpenAI APIs by just changing the base URL and model name in your client code, which makes A/B testing quality much easier.
Does self-hosting improve data privacy compared to cloud APIs?
Only if you own the physical hardware or run within a private VPC you control. Most “self-hosted” deployments use cloud GPU providers where your data still traverses external infrastructure. Anthropic by default does not train on API data and offers BAAs for enterprise customers, so the practical privacy gap for most B2B use cases is smaller than it appears.
What’s the best serving framework for self-hosted LLMs in production?
vLLM is the production standard for throughput-optimized serving — it uses PagedAttention for efficient KV cache management and supports continuous batching. Ollama is better for development and single-user setups. TGI (Text Generation Inference by HuggingFace) is solid for multi-GPU tensor parallel deployments. For most teams running 7B–70B models at production scale, start with vLLM.
Put this into practice
Try the React Performance Optimizer agent — ready to use, no setup required.
Editorial note: API pricing, model capabilities, and tool features change frequently — always verify current details on the vendor’s website before building in production. Code examples are tested at time of writing; pin your dependency versions to avoid breaking changes. Some links in this article may be affiliate links — we may earn a commission if you sign up, at no extra cost to you.