LLM Observability Stack

What it does

Type spike, errors, cost or clean to choose what goes wrong. The page runs a real four step pipeline (find documents, rerank them, write an answer, grade the answer) a dozen times to establish normal behaviour, then runs it again with the fault you picked.

Every step is timed and recorded as a trace. You get the waterfall for one request, showing which step nested inside which and how many milliseconds each one took, plus the token and money totals broken down by model tier, and the p50 and p95 latency for the whole request and for each step.

The last part is the alerting. Latency is compared against a baseline the system learns as it goes, rather than a fixed number someone guessed. Repeat alerts about the same problem are held back by a cooldown and counted, so one incident pages you once instead of forty times, and the page shows you both numbers.

Try it

Output

WHAT WAS RUN
  warm up      12 requests on the healthy pipeline, so the detector learns what normal looks like
  then         5 requests where the answer-writing upstream returns HTTP 503 on every call
  recorded     17 requests, 126 spans, 46 model calls, all measured in this process

ALERTS
  CRITICAL error_rate_high        service=rag       error_rate 0.250 crossed the 0.200 threshold
  1 alert(s) sent across 1 distinct problem(s); 1 repeat(s) held back by the cooldown
  without the cooldown this run would have paged 2 times instead of 1
  problem c40700f3665a still has 1 suppressed repeat(s) inside its cooldown
  the warm up produced no alerts at all, so everything above came from the fault

LATENCY
  whole request   p50 0.7 ms   p95 2.5 ms   p99 2.5 ms   worst 2.5 ms
  step              calls    p50 ms    p95 ms   worst ms
  ------------------------------------------------------
  generate             17       0.2       1.8        1.8
  rerank               17       0.3       0.8        0.8
  judge                12       0.2       0.2        0.2
  retrieve             17       0.1       0.2        0.2

COST AND TOKENS, BY MODEL TIER
  tier              calls  cached   tokens    cost usd
  ----------------------------------------------------
  large                17       4     2752    0.011514
  small                29       8     5344    0.000759
  total 8096 tokens, $0.012273, mean $0.000722 per request
  cache 26% hit rate, $0.005919 of calls avoided
  errors 29% of requests (12% of spans, because one bad step sits next to healthy siblings)
  failures by type: LLMError x15
  most expensive requests: warm-03 $0.001842, warm-05 $0.001832, warm-01 $0.001663

ONE REQUEST, TRACED END TO END
  request fault-00, failed, LLMError: synthetic upstream failure (HTTP 503)
trace f38d043561a34b3b9a8e6e6c05d0621b   total 0.7 ms   spans 6
span                                  start    dur ms  timeline
request [error]                         0.0      0.67  |!!!!!!!!!!!!!!!!!!!!!!!!!!|
  retrieve                              0.0      0.16  | ======                   |
  rerank                                0.2      0.32  |       =============      |
    llm.call                            0.3      0.17  |             =======      |
  generate [error]                      0.5      0.13  |                     !!!!!|
    llm.call [error]                    0.6      0.03  |                        ! |

PRIVACY
  prompts are stored as a salted hash (830d2787baae25ae, 912 characters) and not as text
  spans holding raw prompt text: 0 of 126

Computed live in 27 ms.