Beyond the Token Price

When I look at the token prices frontier labs publish, I wonder how much they tell me about the cost of running an AI product. I can estimate a token bill from how much text is sent to a model and how much it generates. But what happens to the service when most of that work arrives in the same busy hour? These notes come from thinking through that question and digging into a public Azure workload.

I started with a thought experiment: two versions of the same service, using the same model and processing the same number of tokens in a day. In one, requests arrive steadily; in the other, they crowd into a single hour. The token bill could be identical, yet I would expect the second version to need more capacity during that hour, or to make people wait. How much of that difference would show up in real request data?

To find out, I analyzed Microsoft's public Azure trace of code-related inference requests, which records when requests arrived and how long their inputs and outputs were. It covers a week in May 2024. The file has no response times, so I can examine the incoming work but not how quickly Azure completed it.

When requests arrive

My first question was whether an average arrival rate is a reasonable starting point for estimating capacity, or whether it hides too much of the busy periods. Here is the spread after counting the requests in each minute of the trace.

If I replaced these arrivals with a steady stream at the average rate, I would lose the busiest periods. Those are the periods that matter most for the cost question: can the service process incoming work quickly enough, or do requests have to wait? The arrival counts alone cannot tell me whether Azure developed a queue or how many GPUs it needed.

The tradeoff I keep coming back to is how much waiting to accept. Holding enough capacity for the busiest period may leave paid-for computers idle later. Letting requests wait may require less capacity, but only if the delay is acceptable for the product. Without knowing how quickly capacity can change and what response time the service promises, I cannot turn the arrival pattern into a cost estimate.

Input and output lengths

Timing raised a second question: was each request asking for roughly the same amount of work? I separated the text sent to the model from the text it generated, because processing the supplied context and generating an answer are different kinds of work.

If I estimated capacity by repeating one typical request, I would miss how the service handles the longer requests in this file. Some answers take only a few steps of token generation; others take many more. Different input lengths also change how much context processing happens before generation begins. These differences belong in any workload I use to compare services.

I cannot convert these lengths directly into computation or electricity use, because the work each stage requires depends on the model and its serving configuration. Nor can I extend the findings to 128K contexts or long-running agents: the longest input in this file is 7,743 tokens.

Comparing inference services

After looking at timing and length, I still do not know which service would be cheaper. What I have is a workload to compare them on. The missing evidence is how each candidate handles that work under the same response-time requirements, and what it costs to do so.

A public throughput number cannot fill that gap. This MLPerf server result is valid for its particular model, hardware configuration and latency target. It does not tell me how that configuration would perform with the arrivals and request lengths in the Azure trace.

For a comparison I could use, each candidate would process the same requests with the same model and answer-quality target. I would also need to specify how soon the first response should arrive, how quickly the rest of the answer should follow, and how failed requests are handled. Otherwise, I could mistake a slower or less reliable service for a cheaper version of the same one.

The cost I am trying to understand covers both what happens during the rush and what stays paid for afterward. A replay could measure how many requests meet the response-time requirement and how much extra capacity the busy periods need. I would then check which idle resources, failures and retries were actually charged under the purchasing arrangement. None of those results are in the trace.

I also need to be clear about what is being purchased. With a token-billed API, identical token counts may still produce identical bills even if arrival patterns differ, so my comparison would have to account for rate limits, delays and retries separately. With reserved capacity or owned servers, the time paid-for resources sit unused has a more direct effect on cost.

What I still need to measure

I have not replayed this workload on candidate hardware or compared invoices, so I cannot name the cheapest GPU or claim a percentage saving. What I can see is that a comparison built from an average request arriving at an average rate would remove the very variation I set out to investigate.

The next step would be to replay the actual request sequence under common service requirements. Only then could I compare bills for the configurations that meet the required response time, and check whether the choice still holds over another period of the workload. That is the evidence I would look for alongside a published token price.

Data: Microsoft's complete public code-inference file; 16,803,695 requests; May 10–16, 2024; 10,080 minutes; UTC; approximately 692MB. No missing values or negative token counts. The file does not contain charges, hardware configuration, answer quality or observed response latency. AI tools assisted with code and manuscript preparation; numerical results were computed from the source file.

Dataset citation: Jovan Stojkovic, Chaojie Zhang, Íñigo Goiri, Josep Torrellas and Esha Choukse, DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency, HPCA 2025.