Streaming, Backpressure and Resume

What it does

When an assistant types its answer out word by word, there is a small protocol underneath doing the work. This page shows you that protocol: the exact text frames that go down the wire, how long the first word took to arrive, and how evenly the rest followed.

Then it shows the awkward part. The model produces words faster than a slow reader can take them, so something has to hold the difference. The buffer here is deliberately small, and you choose what it does when it fills up: make the model wait, which keeps every word but slows everything down, or throw words away, which stays fast but leaves holes in the answer. You see both, with the cost of each measured.

Last, it kills the connection halfway through and reconnects. The reader tells the server the id of the last word it actually received, and the server carries on from the next one, so the answer arrives complete with nothing repeated.

One honest note: this page shows the protocol and the buffer behaviour, not a live open connection. The server this runs on is a serverless function that cannot hold a stream open, so the frames, the timings and the buffer decisions are produced in memory and printed. Everything you see is real code doing the real thing, just not across a socket.

Try it

Output

STREAMING, BACKPRESSURE AND RESUME  protocol and buffers, no socket
==============================================================================
  you chose           drop
  question            'what should happen when the consumer is slower than the producer'
  answer length       70 tokens, one SSE frame each
  buffer              4 slots between the producer and the reader
  slow reader         3 ms per token, against a producer that needs 0.5 ms

  TRIMMED FOR THIS PAGE: the project's own demo streams the full 277 token
  answer five times over a loopback socket with a 6 ms reader, about ten
  seconds of wall clock. Here the answer is cut to its first 70 tokens and
  the reader is 3 ms per token. The buffer still fills, both policies still
  make their decision, and the resume still replays from the last event id.

------------------------------------------------------------------------------
1. WHAT ACTUALLY GOES DOWN THE WIRE
------------------------------------------------------------------------------
  a fast reader, so nothing is queued. The first frames, exactly as bytes,
  each one closed by a blank line:

    | retry: 750
    | event: open
    | data: {"stream_id": "baseline", "resumed": false, "start_index": 0, "policy": "block", "buffer": 64}
    |
    | id: 0
    | event: token
    | data: A
    |
    | id: 1
    | event: token
    | data:  streaming
    |
    | id: 2
    | event: token
    | data:  inference
    |
    | ...

  and the closing frame
    | id: 69
    | event: done
    | data: {"tokens": 70, "delivered_from": 0, "dropped": 0}
    |

  fed back through the project's own parser: 72 frames read, 70 of them tokens
  the id on each frame is the token index. That is the only reason a resume
  can work: the reader can name exactly how far it got.

------------------------------------------------------------------------------
2. TIMING  a fast reader, nothing in the way
------------------------------------------------------------------------------
  time to first token   0.78 ms
  whole answer          41.07 ms
  share spent waiting for the first token   1.9 percent
  between tokens        p50 0.58 ms, p95 0.63 ms, worst 0.66 ms
  throughput            1714 tokens per second
  time to first token is the number a reader can feel. An answer that takes
  four seconds but starts printing in 200 ms reads as fast; one that arrives
  all at once after two seconds of a spinner reads as slow.
  p95 is tracked separately from the average because one 900 ms stall in the
  middle of an otherwise smooth stream is what makes a stream look broken.

------------------------------------------------------------------------------
3. THE BUFFER UNDER A SLOW READER  4 slots, 3 ms per token
------------------------------------------------------------------------------
  policy    delivered  dropped  gap runs  high water  producer waited   wall ms
  -----------------------------------------------------------------------------
  block        70 of 70        0         0      4 of 4          162 ms       218
  drop         17 of 70       53        13      4 of 4            0 ms        53

  under drop the loss is announced, never silent. A gap frame looks like:
    | id: 5
    | event: gap
    | data: {"dropped": 1}
    |
  a dropped token nobody is told about is a silently truncated answer.
  block kept all 70 tokens and took 218 ms, waiting 162 ms across 65 pauses.
  drop finished in 53 ms and lost 53 tokens in 13 runs.
  neither is free. An unbounded queue looks like block with no wait and no
  loss, right up to the point where one slow reader per stream turns the
  difference in speed into resident memory and the process is killed.
  block is right for an answer someone is reading. drop is only right when
  the payload is refreshable state, a progress percentage or a preview.

------------------------------------------------------------------------------
4. DISCONNECT AND RESUME  the connection dies after 18 tokens
------------------------------------------------------------------------------
  first connection      18 tokens, last event id 17, completed=False
  producer stopped      the server generated 19 tokens then saw the
                        cancel and stopped, instead of generating all 70 into a dead connection
  reconnect sends       Last-Event-ID: 17
  second connection     resumed at index 18, 52 more tokens, completed=True

  event ids received    [0, 1, 2, 3] ... [66, 67, 68, 69]
  contiguous 0..69       True
  duplicated ids        none
  text matches a stream that was never interrupted   True
  the resume is served out of the cached token list, so the model is not run
  a second time. Re-running it would bill the caller twice, and at any
  temperature above zero the continuation would not match what the reader
  already has on screen.

------------------------------------------------------------------------------
  Nothing above opened a connection. The SSE frames were built by the
  project's own formatter and read back by its own parser, the buffer is the
  project's BoundedChannel, and the producer is the project's own thread body
  including the cancellation check. A serverless function cannot hold a
  stream open, so the protocol is shown rather than performed.

Computed live in 400 ms.