See what a token stream really sends, what happens when the reader is too slow, and how a dropped connection picks up where it left off.
When an assistant types its answer out word by word, there is a small protocol underneath doing the work. This page shows you that protocol: the exact text frames that go down the wire, how long the first word took to arrive, and how evenly the rest followed.
Then it shows the awkward part. The model produces words faster than a slow reader can take them, so something has to hold the difference. The buffer here is deliberately small, and you choose what it does when it fills up: make the model wait, which keeps every word but slows everything down, or throw words away, which stays fast but leaves holes in the answer. You see both, with the cost of each measured.
Last, it kills the connection halfway through and reconnects. The reader tells the server the id of the last word it actually received, and the server carries on from the next one, so the answer arrives complete with nothing repeated.
One honest note: this page shows the protocol and the buffer behaviour, not a live open connection. The server this runs on is a serverless function that cannot hold a stream open, so the frames, the timings and the buffer decisions are produced in memory and printed. Everything you see is real code doing the real thing, just not across a socket.
STREAMING, BACKPRESSURE AND RESUME protocol and buffers, no socket
==============================================================================
you chose disconnect
question 'what should happen when the consumer is slower than the producer'
answer length 70 tokens, one SSE frame each
buffer 4 slots between the producer and the reader
slow reader 3 ms per token, against a producer that needs 0.5 ms
TRIMMED FOR THIS PAGE: the project's own demo streams the full 277 token
answer five times over a loopback socket with a 6 ms reader, about ten
seconds of wall clock. Here the answer is cut to its first 70 tokens and
the reader is 3 ms per token. The buffer still fills, both policies still
make their decision, and the resume still replays from the last event id.
------------------------------------------------------------------------------
1. WHAT ACTUALLY GOES DOWN THE WIRE
------------------------------------------------------------------------------
a fast reader, so nothing is queued. The first frames, exactly as bytes,
each one closed by a blank line:
| retry: 750
| event: open
| data: {"stream_id": "baseline", "resumed": false, "start_index": 0, "policy": "block", "buffer": 64}
|
| id: 0
| event: token
| data: A
|
| id: 1
| event: token
| data: streaming
|
| id: 2
| event: token
| data: inference
|
| ...
and the closing frame
| id: 69
| event: done
| data: {"tokens": 70, "delivered_from": 0, "dropped": 0}
|
fed back through the project's own parser: 72 frames read, 70 of them tokens
the id on each frame is the token index. That is the only reason a resume
can work: the reader can name exactly how far it got.
------------------------------------------------------------------------------
2. TIMING a fast reader, nothing in the way
------------------------------------------------------------------------------
time to first token 0.86 ms
whole answer 41.24 ms
share spent waiting for the first token 2.1 percent
between tokens p50 0.58 ms, p95 0.62 ms, worst 0.70 ms
throughput 1711 tokens per second
time to first token is the number a reader can feel. An answer that takes
four seconds but starts printing in 200 ms reads as fast; one that arrives
all at once after two seconds of a spinner reads as slow.
p95 is tracked separately from the average because one 900 ms stall in the
middle of an otherwise smooth stream is what makes a stream look broken.
------------------------------------------------------------------------------
3. THE BUFFER UNDER A SLOW READER 4 slots, 3 ms per token
------------------------------------------------------------------------------
policy delivered dropped gap runs high water producer waited wall ms
-----------------------------------------------------------------------------
block 70 of 70 0 0 4 of 4 162 ms 218
drop 17 of 70 53 13 4 of 4 0 ms 54
block kept all 70 tokens and took 218 ms, waiting 162 ms across 65 pauses.
drop finished in 54 ms and lost 53 tokens in 13 runs.
neither is free. An unbounded queue looks like block with no wait and no
loss, right up to the point where one slow reader per stream turns the
difference in speed into resident memory and the process is killed.
block is right for an answer someone is reading. drop is only right when
the payload is refreshable state, a progress percentage or a preview.
------------------------------------------------------------------------------
4. DISCONNECT AND RESUME the connection dies after 18 tokens
------------------------------------------------------------------------------
first connection 18 tokens, last event id 17, completed=False
producer stopped the server generated 19 tokens then saw the
cancel and stopped, instead of generating all 70 into a dead connection
reconnect sends Last-Event-ID: 17
second connection resumed at index 18, 52 more tokens, completed=True
event ids received [0, 1, 2, 3] ... [66, 67, 68, 69]
contiguous 0..69 True
duplicated ids none
text matches a stream that was never interrupted True
the resume is served out of the cached token list, so the model is not run
a second time. Re-running it would bill the caller twice, and at any
temperature above zero the continuation would not match what the reader
already has on screen.
------------------------------------------------------------------------------
Nothing above opened a connection. The SSE frames were built by the
project's own formatter and read back by its own parser, the buffer is the
project's BoundedChannel, and the producer is the project's own thread body
including the cancellation check. A serverless function cannot hold a
stream open, so the protocol is shown rather than performed.