LLM Observability Stack

What it does

Type spike, errors, cost or clean to choose what goes wrong. The page runs a real four step pipeline (find documents, rerank them, write an answer, grade the answer) a dozen times to establish normal behaviour, then runs it again with the fault you picked.

Every step is timed and recorded as a trace. You get the waterfall for one request, showing which step nested inside which and how many milliseconds each one took, plus the token and money totals broken down by model tier, and the p50 and p95 latency for the whole request and for each step.

The last part is the alerting. Latency is compared against a baseline the system learns as it goes, rather than a fixed number someone guessed. Repeat alerts about the same problem are held back by a cooldown and counted, so one incident pages you once instead of forty times, and the page shows you both numbers.

Try it

Output

WHAT WAS RUN
  warm up      12 requests on the healthy pipeline, so the detector learns what normal looks like
  then         4 requests where a retry loop repeats the evidence 40 times and inflates the prompt
  recorded     16 requests, 128 spans, 48 model calls, all measured in this process

ALERTS
  WARNING  llm_latency_spike      step=generate     latency_ms 6.4 is 74.5 sigma from a baseline of 1.6 (threshold 3.0 sigma)
  WARNING  cost_per_request_high  tenant=acme       cost_usd 0.021 crossed the 0.020 threshold
  2 alert(s) sent across 2 distinct problem(s); 0 repeat(s) held back by the cooldown
  the warm up produced no alerts at all, so everything above came from the fault
  the sigma number is enormous because a sub-millisecond baseline has almost no spread; the detection is reliable, the size of it is not

LATENCY
  whole request   p50 0.9 ms   p95 7.2 ms   p99 7.2 ms   worst 7.2 ms
  step              calls    p50 ms    p95 ms   worst ms
  ------------------------------------------------------
  generate             16       0.3       6.5        6.5
  rerank               16       0.3       0.8        0.8
  judge                16       0.2       0.2        0.2
  retrieve             16       0.1       0.2        0.2

COST AND TOKENS, BY MODEL TIER
  tier              calls  cached   tokens    cost usd
  ----------------------------------------------------
  large                16       4    27421    0.088521
  small                32       8     5557    0.000816
  total 32978 tokens, $0.089337, mean $0.005584 per request
  cache 25% hit rate, $0.005919 of calls avoided
  errors 0% of requests (0% of spans, because one bad step sits next to healthy siblings)
  most expensive requests: fault-01 $0.020709, fault-02 $0.019628, fault-00 $0.018928

ONE REQUEST, TRACED END TO END
  request fault-03, served
trace a93dd7bccaac4862a016b84b6b5f68b9   total 6.4 ms   spans 8
span                                  start    dur ms  timeline
request                                 0.0      6.37  |==========================|
  retrieve                              0.0      0.15  |=                         |
  rerank                                0.2      0.30  | =                        |
    llm.call                            0.3      0.15  | =                        |
  generate                              0.5      5.62  |  ======================= |
    llm.call                            2.3      3.83  |         ================ |
  judge                                 6.2      0.21  |                         =|
    llm.call                            6.2      0.11  |                          |

PRIVACY
  prompts are stored as a salted hash (830d2787baae25ae, 912 characters) and not as text
  spans holding raw prompt text: 0 of 128

Computed live in 48 ms.