Self-hosting an LLM or using an API: what it costs
By TryRAG · Updated
Try it: the Inference Cost Calculator compares a GPU you rent with an API price and finds the break-even point.
There are two ways to pay for a language model. An API charges for each token. A GPU you rent, or own, charges for each hour, whether it is answering questions or waiting for them. Which is cheaper depends less on the prices than on how busy the GPU is.
The cost of a self-hosted token
Cost per 1M tokens = GPU price per hour ÷ (tokens per second × 3,600 × utilization) × 1,000,000
Tokens per second is what the server produces when fully busy, across all the requests it handles at once. Utilization is the share of the time it is that busy. Take a GPU at $3.99 an hour producing 1,000 tokens a second:
| Utilization | Cost per 1M tokens |
|---|---|
| 100% | $1.11 |
| 50% | $2.22 |
| 25% | $4.43 |
| 10% | $11.08 |
The same GPU, at the same price, is ten times dearer per token when it is idle nine-tenths of the time. A service used during office hours in one country is rarely above 30% across a month.
The cost of an API token
APIs price input and output separately. For every answer you also pay for the whole prompt. With a 1,000-token prompt and a 500-token answer, at $0.75 per million input tokens and $4.50 per million output tokens, each million answer tokens costs $4.50 plus two million prompt tokens at $0.75, which is $6.00. In RAG the prompt is usually several times longer than the answer, so the input price counts for more than it looks. The Token Counter shows current prices.
The break-even point
The GPU costs a fixed amount each month: $3.99 × 730 hours is about $2,913. Divide that by the API price per million tokens to get the volume at which the two cost the same. At $6.00 that is about 485 million answer tokens a month, which is 18% of what the GPU could produce. Below that, the API is cheaper. Above it, self-hosting is.
What moves the answer
- Throughput. Serving many requests in each step multiplies the tokens per second. See continuous batching.
- Model size. A smaller or quantized model runs faster and fits a cheaper GPU.
- Prompt length. Shorter RAG prompts lower the API bill directly, and raise self-hosted throughput.
- Traffic shape. Steady traffic keeps utilization high. Short peaks with long quiet periods do not.
What the sum leaves out
- People. Someone has to set up, watch and update the servers. For a small team this often outweighs the GPU bill.
- Spare capacity. A service that must always answer needs more than one GPU, and headroom for peaks.
- Discounts on both sides. Reserved and spot GPUs are cheaper than on-demand; APIs offer prompt caching and batch prices.
- Quality. The comparison assumes the two models are equally good for your task. Check that first.
- Reasons other than cost. Keeping data on your own machines, or fixing a model version, can decide the question on their own.
Words you will meet
- Utilization: the share of the time a GPU is producing tokens at its full rate.
- Throughput: tokens per second across all requests.
- On-demand: rented by the hour with no commitment.
- Break-even: the volume at which two options cost the same.