LLM Observability Stack

What it does

Type spike, errors, cost or clean to choose what goes wrong. The page runs a real four step pipeline (find documents, rerank them, write an answer, grade the answer) a dozen times to establish normal behaviour, then runs it again with the fault you picked.

Every step is timed and recorded as a trace. You get the waterfall for one request, showing which step nested inside which and how many milliseconds each one took, plus the token and money totals broken down by model tier, and the p50 and p95 latency for the whole request and for each step.

The last part is the alerting. Latency is compared against a baseline the system learns as it goes, rather than a fixed number someone guessed. Repeat alerts about the same problem are held back by a cooldown and counted, so one incident pages you once instead of forty times, and the page shows you both numbers.

Try it

Output

WHAT WAS RUN
  warm up      12 requests on the healthy pipeline, so the detector learns what normal looks like
  then         5 requests where nothing is broken at all
  recorded     17 requests, 136 spans, 51 model calls, all measured in this process

ALERTS
  WARNING  llm_latency_spike      step=generate     latency_ms 0.5 is 3.5 sigma from a baseline of 0.4 (threshold 3.0 sigma)
  1 alert(s) sent across 1 distinct problem(s); 0 repeat(s) held back by the cooldown
  the sigma number is enormous because a sub-millisecond baseline has almost no spread; the detection is reliable, the size of it is not

LATENCY
  whole request   p50 0.9 ms   p95 1.1 ms   p99 1.1 ms   worst 1.1 ms
  step              calls    p50 ms    p95 ms   worst ms
  ------------------------------------------------------
  generate             17       0.3       0.5        0.5
  rerank               17       0.3       0.3        0.3
  judge                17       0.2       0.2        0.2
  retrieve             17       0.1       0.1        0.1

COST AND TOKENS, BY MODEL TIER
  tier              calls  cached   tokens    cost usd
  ----------------------------------------------------
  large                17       4     3910    0.018648
  small                34       8     5833    0.000873
  total 9743 tokens, $0.019521, mean $0.001148 per request
  cache 24% hit rate, $0.005919 of calls avoided
  errors 0% of requests (0% of spans, because one bad step sits next to healthy siblings)
  most expensive requests: fault-03 $0.001856, warm-03 $0.001842, warm-05 $0.001832

ONE REQUEST, TRACED END TO END
  request fault-04, served
trace 5c4a58a05f0443dda47d4ce919385db4   total 1.0 ms   spans 8
span                                  start    dur ms  timeline
request                                 0.0      0.98  |==========================|
  retrieve                              0.0      0.10  |===                       |
  rerank                                0.1      0.29  |   ========               |
    llm.call                            0.3      0.14  |       ====               |
  generate                              0.4      0.32  |            ========      |
    llm.call                            0.5      0.22  |              ======      |
  judge                                 0.8      0.21  |                    ======|
    llm.call                            0.9      0.11  |                       ===|

PRIVACY
  prompts are stored as a salted hash (830d2787baae25ae, 912 characters) and not as text
  spans holding raw prompt text: 0 of 136

Computed live in 18 ms.