Streaming LLM responses without losing track of the cost

To stream a response from an LLM API, set stream to true on the request and read the reply as server-sent events: a sequence of data: lines, each carrying a small JSON chunk of the answer, closed by data: [DONE]. What streaming makes harder is tracking cost, because the token usage a bill is calculated from is not in those chunks; on the OpenAI contract it arrives in one extra chunk at the end, only if you asked for it, and a stream that is cut off may never deliver it.

P/9 answers the cost half in rupees rather than tokens. Send X-P9-Debug-Metrics: true on a streamed request and the gateway appends the cost of that request after the end marker, and every request it serves, streamed or not, lands in the request log with its settled cost.

A stream is a series of small events, not one slow response

With stream set to true, the connection stays open while the model generates. Each event is a data: line holding one chat.completion.chunk, events are separated by a blank line, and the text arrives in choices[0].delta.content a few characters at a time. A final chunk carries the finish_reason: stop if the model finished, length if max_tokens cut it off.

The stream ends with data: [DONE], which is not JSON and is the signal to stop parsing. Every OpenAI SDK handles the framing for you. Without one, split on blank lines, keep the lines that start with data:, stop at [DONE], and never assume one network read holds one whole event, because an event can arrive split across two.

What streaming buys is time to first token: the reader sees the first words while the rest is still being generated. What it costs is that the accounting moves to the end of the stream.

Why cost is the part streaming hides

A non-streamed response carries a usage object beside the answer: prompt tokens, completion tokens and a total. A streamed response has nowhere obvious to put that, because usage is only known once generation stops, and by then every chunk of text has already gone.

The OpenAI contract solves it with an opt-in. Send stream_options with include_usage set to true and one more chunk is streamed before data: [DONE], with usage for the whole request and an empty choices list. Every chunk before it carries usage as null, and without the option there is no usage in the stream at all.

Two things then break in practice. A loop written to print text reads choices[0] on every chunk, so it crashes on the usage chunk, whose list is empty. And a loop written to print text rarely looks at anything else, so the usage arrives and is thrown away.

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" today."},"finish_reason":null}],"usage":null}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":null}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":38,"completion_tokens":21,"total_tokens":59}}

data: [DONE]

The stream that stops early is the one you most need to see

Streams end early for ordinary reasons: a user closes the tab or presses stop, a connection drops, a timeout fires, or the model server drops the stream mid-way. OpenAI's own API reference says it plainly: if the stream is interrupted, you may not receive the final usage chunk.

That leaves client-side accounting blind in exactly the wrong place. A cost record assembled from what clients received counts the requests that went well and misses the ones that did not, so an incident that shows up as a burst of abandoned streams looks cheaper in your own records than it was.

On P/9 the gateway records how every request ended: client_disconnect when your side closed the connection, upstream_stream_aborted when the model server dropped it, proxy_guillotine when the gateway's own timeout cut it off. Tokens relayed before the stop are billed and nothing after it is. Treat a disconnect on your side as finished rather than retryable, because a retried chat completion is a second, billed generation.

Rate limits and rupee spend caps that stop a retry loop becoming a bill

What to record for every request, streamed or not

Record cost as the figure the provider settled, not as tokens to multiply later. A rate table copied into your code is right on the day it is written and wrong from the first price change, and recomputing old requests against a new table quietly rewrites what last month cost.

Beyond that, the record worth keeping per request is short, and it is the minimum that answers what a feature costs, why one day was expensive, and whether an incident was also a spend incident.

• Your own request id, sent to the provider as a tag, so its log and yours join on one value
• The model or route id, and whether the call streamed
• Input and output token counts, wherever the response reports them
• The cost as the provider computed it, in the currency you are billed in
• How the request ended: finished, cut off by max_tokens, disconnected by the client, or failed upstream
• Time to first token and total time
• The key, feature and customer the call belongs to

How input and output rates turn a workload into a rupee figure

How P/9 puts the cost of a streamed request inside the stream

Send X-P9-Debug-Metrics: true on a streaming request and, after data: [DONE], the gateway appends one more event holding a k2i_metrics object: ttft_ms, the time from the request reaching the gateway to the first token leaving it; tok_s, output tokens per second; and cost_micro_pkr, what that request cost in micro-rupees. Divide by a million for rupees, so 45120 is Rs 0.04512.

It is streaming only, and the header has no effect on a non-streamed request. Because the event arrives after the end marker, stock OpenAI clients never show it: the official Python SDK stops reading at [DONE]. That makes the header safe to send everywhere, and it means reading the figure takes a small reader of your own, such as this one.

import json
import os

import httpx

cost_micro_pkr = None

with httpx.stream(
    "POST",
    "https://api.paranine.com/v1/chat/completions",
    headers={
        "Authorization": "Bearer " + os.environ["P9_API_KEY"],
        "X-P9-Debug-Metrics": "true",
    },
    json={
        "model": "paranine/gpt-oss-120b(Global)",
        "stream": True,
        "messages": [{"role": "user", "content": "Summarise this ticket in one line."}],
    },
    timeout=120,
) as response:
    response.raise_for_status()
    for line in response.iter_lines():
        if not line.startswith("data:"):
            continue
        data = line[5:].strip()
        if data == "[DONE]":
            continue  # keep reading: the metrics event comes after it
        event = json.loads(data)
        if "k2i_metrics" in event:
            cost_micro_pkr = event["k2i_metrics"]["cost_micro_pkr"]
        elif event.get("choices"):
            delta = event["choices"][0].get("delta", {})
            print(delta.get("content") or "", end="", flush=True)

print()
if cost_micro_pkr is None:
    print("No metrics event: find this request in the request log")
else:
    print(f"Cost: Rs {cost_micro_pkr / 1_000_000:.6f}")

The request log covers the requests the stream cannot

A non-streamed request carries the standard usage object in its response body, and on P/9 that is what a synchronous call is billed on. The debug header does nothing there, and a stream that never reached its end never delivered its event, so for both the rupee figure is in the request log.

The log holds a row for every request the gateway serves: route, key and project, status, whether it streamed, how it ended, timings, the settled cost in rupees, and any X-P9-Meta- headers you sent, such as X-P9-Meta-feature_tag: ticket_summary. Prompts and completions are not part of it. Search it by request id or tag value on the dashboard's Request logs page or through GET /api/analytics/requests, and filter by termination reason or minimum cost, which is how a burst of abandoned streams becomes a list.

For the total rather than the row, each key's panel on the dashboard shows that key's spend for the window, which is the quickest answer to what one service costs.

Why cost accounting belongs in the gateway rather than in every service

The short version

Stream for time to first token, and decide where cost is recorded before you ship, because a streamed response will not volunteer it. Where the API offers a usage chunk, ask for it and guard the loop against its empty choices, and treat anything assembled on the client as an undercount of the requests that went wrong.

On P/9, read cost_micro_pkr from the event after data: [DONE] when you want the figure inside your own code, and the request log when you want it for every request, including the ones that never finished.