Self-hosting an LLM or using an API: what it costs

By TryRAG · Updated

Try it: the Inference Cost Calculator compares a GPU you rent with an API price and finds the break-even point.

There are two ways to pay for a language model. An API charges for each token. A GPU you rent, or own, charges for each hour, whether it is answering questions or waiting for them. Which is cheaper depends less on the prices than on how busy the GPU is.

The cost of a self-hosted token

Cost per 1M tokens = GPU price per hour ÷ (tokens per second × 3,600 × utilization) × 1,000,000

Tokens per second is what the server produces when fully busy, across all the requests it handles at once. Utilization is the share of the time it is that busy. Take a GPU at $3.99 an hour producing 1,000 tokens a second:

UtilizationCost per 1M tokens
100%$1.11
50%$2.22
25%$4.43
10%$11.08

The same GPU, at the same price, is ten times dearer per token when it is idle nine-tenths of the time. A service used during office hours in one country is rarely above 30% across a month.

The cost of an API token

APIs price input and output separately. For every answer you also pay for the whole prompt. With a 1,000-token prompt and a 500-token answer, at $0.75 per million input tokens and $4.50 per million output tokens, each million answer tokens costs $4.50 plus two million prompt tokens at $0.75, which is $6.00. In RAG the prompt is usually several times longer than the answer, so the input price counts for more than it looks. The Token Counter shows current prices.

The break-even point

The GPU costs a fixed amount each month: $3.99 × 730 hours is about $2,913. Divide that by the API price per million tokens to get the volume at which the two cost the same. At $6.00 that is about 485 million answer tokens a month, which is 18% of what the GPU could produce. Below that, the API is cheaper. Above it, self-hosting is.

What moves the answer

What the sum leaves out

Words you will meet