Token Pricing and AI Clouds

AI cloud providers are seeing a shift in how customers purchase AI inference, as more and more customers move from per-GPU-hour billing to per-token billing. While this transition may seem trivial, it has significant implications for the economics of AI cloud providers. When customers rent GPUs, they pay for the GPU regardless of whether it is being used, but when they purchase tokens, they are only charged for the output, so the idle time of the GPUs is absorbed by the cloud provider.

This gives token providers an incentive to make their GPUs produce the most tokens possible while maintaining utilization. If customers can readily move between providers that are all running the same model, efficiency becomes an important competitive factor. Furthermore, since customers are buying model access rather than chip access, the infrastructure can be switched out from under them, allowing more efficient clouds to compete and paving the way for non-NVIDIA chips.

Idle GPUs

Renting GPUs requires customers to pay per hour for the time that a GPU is reserved for them, whereas purchasing inference through an API means customers are charged per token, where tokens are small pieces of text (often partial words) that go into and come out of the model. Token pricing also requires inference providers to translate their infrastructure into model output before they can start charging. Both Together and Fireworks offer GPU-hour and token pricing, with Together offering private deployments priced per GPU hour and a public API priced per token [1], and Fireworks offering a similar model [2]. In both cases the provider hosts the model for the customer, so the distinction is whether the customer is responsible for hardware downtime.

If the customer has leased a private deployment, they continue to be charged hourly regardless of how many requests they send while the model runs, even if they leave it running overnight. If they are using a public token API with no minimums, however, their bill decreases as the number of tokens they use decreases, so it is up to the inference provider to keep its hardware busy or pay for the downtime. GPU rental companies face the same cost for machines that aren't rented, but once a machine is rented, the customer pays for its idle time. Table 1 summarizes who absorbs the cost of idle hardware under each pricing model.

Table 1. Common inference pricing models, and the party that absorbs the cost of idle hardware.

Token providers are in some ways similar to compute traders, since they buy or reserve GPU time, run models on it, and sell the output per token, making a profit when they generate more tokens than expected and losing money when their GPUs are not being utilized.

To provide an example, assume a deployment costs $10 an hour to run. If the deployment processes 10,000 requests in an hour, the cost is a tenth of a cent per request, but if it only processes 5,000 requests in that hour, the cost per request doubles. These are illustrative numbers that do not include any other costs in the business model, but they demonstrate how a provider's token cost is a function of both the throughput capability of the GPU and its utilization.

Serving software and token costs

How many tokens each GPU can process is largely a function of the serving software. Models generate text token by token, and when a GPU runs multiple requests at the same time, those requests can share each read of the model weights from memory, a technique known as batching that can drastically increase total throughput, depending on the model and batch size [6]. In practice, however, batching is difficult because requests arrive and leave at different times, and some may only require one line of text while others require many pages.

Orca partially solved this by allowing the batch to be updated after each token, so a request that has finished can immediately leave the batch and be replaced with a new one rather than waiting until the entire batch is complete [7].

Memory has a similar effect, since every active request keeps a cache of attention values for the text processed so far, and if that memory is used inefficiently, fewer requests can run on a GPU at once. vLLM stores this cache in small blocks to reduce memory waste, and in its original 2023 measurements, vLLM showed a two- to four-fold throughput improvement over competing systems at the same latency [8]. How much of an increase this leads to depends on the specific configuration, but the increase demonstrates how much the software stack can affect GPU performance.

Performance changes can be that dramatic even from tweaking a few configuration options, as one engineer reported more than tripling throughput, from 411 to 1,366 tokens per second, by changing four configuration options on a single B200 without writing any new GPU code [9]. At an illustrative price of $5 an hour, the first case works out to about 1.5 million tokens per hour, or about $3.40 per million tokens, while the second works out to about 4.9 million tokens per hour, or about $1.00 per million tokens, so the same GPU at the same price can produce tokens at about a third of the cost.

An increase of this magnitude has many implications, since with token pricing, an inference provider can serve the same traffic with less infrastructure and either keep the difference or lower prices for customers. Customers operating their own GPU infrastructure can make the same improvements, but in the cloud it is a one-time change that benefits every customer.

There is a limit to how far a provider can push throughput, however, because at some point latency increases for individual requests, and customers will notice the degradation if a provider focuses purely on raw throughput. Providers therefore need to compare performance at the same latency rather than comparing raw throughput alone. DistServe uses the term "goodput" for the number of requests a system can process within a given latency target, and goodput can be improved partly by decoupling the prefill and decode stages so that they can run on different hardware [10].

Traffic volume and batching

There also has to be enough demand in the system to run the hardware efficiently. If multiple customers use the same model and are busy at different times of the day, they can save money by sharing the underlying hardware rather than each buying hardware for its own peak, and AlpaServe researches how to share compute resources across models to handle this uneven demand, which introduces an additional communication cost [11].

However, simply increasing the number of users does not increase system efficiency, because it is more difficult to handle a small volume of queries spread over hundreds of models than a steady stream of queries over a few models. Since every model requires memory to store its weights, requests for one model cannot share weight reads with requests for another, so MuxServe investigates how an inference provider can allocate and share models based on their popularity [12].

This shows the competitive edge a cloud gains at larger scale, since with higher demand for the right models, a provider can batch requests more efficiently and fill idle time, lowering costs that can then be passed on to customers as lower prices. Having more GPUs is not the only variable, however, because an inference provider can optimize its serving stack to the extreme and still fail to reach cost efficiency without enough customers to fill its hardware.

Competition and infrastructure considerations

Token pricing also makes inference providers easy to compare. If multiple providers host the same open model, customers can compare and switch between them on token price, latency, and reliability, because the model and the API are the same, and OpenRouter is an example of this, hosting the same model from multiple providers under one API and letting customers route their requests based on cost, throughput, or latency [13].

Token pricing therefore encourages providers to compete on price and efficiency, and specifically on which provider can serve at the lowest cost for a given latency. The cheapest token is not necessarily the same product, however, because inference providers can host the same model at different quantization levels, and OpenRouter lets customers filter providers by quantization level, from 4-bit to full 32-bit [13]. Quantization can reduce costs, but it can also reduce the quality of the model's output.

If a price-only market develops, prices will drop continuously as quality is sacrificed, so it is important to be able to compare model versions, quantization, latency, and data policy in addition to price.

Splitting the prefill and decode parts of inference also enables different infrastructure to be used for each, since prefill is typically compute-bound and decode more bandwidth-bound (particularly for small batch sizes), so one type of chip may not be optimal for both [6]. However, this is not a one-size-fits-all solution, as inference providers and workloads vary.

Gimlet Labs has already experimented with using NVIDIA B200s for prefill and Intel Gaudi 3 chips for decode, providing cost estimates for a range of scenarios and latency targets based on its own benchmarks and models [14]. Mixing hardware may not always be cheaper, but the work demonstrates how a provider can use different hardware for different stages of a request.

Token sales make it easier for providers to adopt newer chips, because customers calling a model API never touch the hardware. The inference provider is in charge of performance and can change accelerators without affecting the model or its performance guarantees, and even if this requires engineering work that sometimes ends up costing more than it saves, the provider only needs to do it once for all of its customers, while lower token costs on new hardware make it a more attractive option for customers using the same model.

Inference providers can also encourage customers to use them more efficiently by offering discounts. If many customers begin their prompts with the same text, the provider can reuse the computation for that part rather than recomputing it for every request, which vLLM supports through prefix caching [15], and Fireworks offers a discount on cached input tokens to share some of the benefit [3]. Providers can also offer discounts for slower inference, as Together offers batch prices up to 50% off its regular prices for jobs that can wait, though as of this writing only certain models are listed as eligible [5].

In exchange, the inference provider gains flexibility in how it schedules requests, so while newer chips are one way for inference providers to charge less, accepting longer latency is another. Some customers will still prefer dedicated GPUs, either because they have steady enough traffic to keep the hardware busy and undercut per-token prices, or because they want to use their own model or have more control, and Together offers dedicated inference as well, through the same API as its shared inference service [1].

It is hard to draw a strict line between these two approaches, since Fireworks sells reserved throughput, where customers commit to spending a certain amount per minute, their token usage is deducted from that amount each minute, and anything they don't use is lost [4]. The billing is still per token, but customers take back some of the risk of paying for unused capacity, so while the invoice still shows tokens, the contract decides who carries the risk.

General thoughts

Inference clouds will likely specialize, with some customers wanting fast inference and others willing to wait for a better price, so the lowest token price on a provider's website won't be a good measure of how well its business is doing. The main thing token pricing does is shift the cost of unused resources to the inference provider, which ties the provider's margins to how efficiently it generates tokens from each GPU and how fully it uses them, so inference efficiency will likely become a more important part of how AI clouds compete.

The most efficient providers can charge less, be faster, or make more money, and how much money they make will depend on how many other providers offer the same model. For inference customers, GPU count will matter less than how well a provider can run the required model while meeting performance, quality, and cost goals, which gives inference providers an incentive to improve their software and to explore new chips beyond the ones their customers know.

Find my more sporadic thoughts on this on twitter, including on inference providers as partly compute traders, on one B200 going from 411 to 1,366 tokens per second, on OpenRouter as a sign of a token market forming, on why comparing providers only by price is a bad way to build that market, on why inference providers have to choose between fast and cheap, and on inference being priced by time of day.

References

1. Together AI. "Dedicated Endpoints Overview." Together AI Documentation. https://docs.together.ai/docs/dedicated-endpoints/overview
2. Fireworks AI. "Pricing." https://fireworks.ai/pricing
3. Fireworks AI. "Serverless Pricing." Fireworks AI Documentation. https://docs.fireworks.ai/serverless/pricing
4. Fireworks AI. "Reserved Throughput." Fireworks AI Documentation. https://docs.fireworks.ai/serverless/reserved-throughput
5. Together AI. "Batch Inference Overview." Together AI Documentation. https://docs.together.ai/docs/inference/batch/overview
6. Austin, J., et al. "All About Transformer Inference." How to Scale Your Model (the Scaling Book), Google DeepMind. https://jax-ml.github.io/scaling-book/inference/
7. Yu, G.-I., et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." OSDI 2022. https://www.usenix.org/conference/osdi22/presentation/yu
8. Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." arXiv:2309.06180, 2023. https://arxiv.org/abs/2309.06180
9. Jaber (@Akashi203). Post on X, August 6, 2026. https://x.com/Akashi203/status/2085521067603316767
10. Zhong, Y., et al. "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI 2024. https://arxiv.org/abs/2401.09670
11. Li, Z., et al. "AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving." OSDI 2023. https://www.usenix.org/conference/osdi23/presentation/li-zhouhan
12. Duan, J., et al. "MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving." ICML 2024. https://proceedings.mlr.press/v235/duan24a.html
13. OpenRouter. "Provider Routing." OpenRouter Documentation. https://openrouter.ai/docs/features/provider-routing
14. Gimlet Labs. "Splitting LLM Inference Across Different Hardware Platforms." October 13, 2025. https://gimletlabs.ai/blog/multivendor-prefill-decode-disaggregation
15. vLLM. "Automatic Prefix Caching." vLLM Documentation. https://docs.vllm.ai/en/latest/design/prefix_caching/