Break a traced AI pipeline on purpose and see the traces, the bill, the latency and the alerts it produced.
Type spike, errors, cost or clean to choose what goes wrong. The page runs a real four step pipeline (find documents, rerank them, write an answer, grade the answer) a dozen times to establish normal behaviour, then runs it again with the fault you picked.
Every step is timed and recorded as a trace. You get the waterfall for one request, showing which step nested inside which and how many milliseconds each one took, plus the token and money totals broken down by model tier, and the p50 and p95 latency for the whole request and for each step.
The last part is the alerting. Latency is compared against a baseline the system learns as it goes, rather than a fixed number someone guessed. Repeat alerts about the same problem are held back by a cooldown and counted, so one incident pages you once instead of forty times, and the page shows you both numbers.
WHAT WAS RUN
warm up 12 requests on the healthy pipeline, so the detector learns what normal looks like
then 5 requests where the answer-writing step suddenly takes 30 ms instead of a fraction of one
recorded 17 requests, 136 spans, 51 model calls, all measured in this process
ALERTS
WARNING llm_latency_spike step=generate latency_ms 30.6 is 1313.2 sigma from a baseline of 6.4 (threshold 3.0 sigma)
1 alert(s) sent across 1 distinct problem(s); 0 repeat(s) held back by the cooldown
the warm up produced no alerts at all, so everything above came from the fault
the sigma number is enormous because a sub-millisecond baseline has almost no spread; the detection is reliable, the size of it is not
LATENCY
whole request p50 0.9 ms p95 31.6 ms p99 31.6 ms worst 31.6 ms
step calls p50 ms p95 ms worst ms
------------------------------------------------------
generate 17 0.3 30.7 30.7
judge 17 0.2 0.5 0.5
rerank 17 0.3 0.4 0.4
retrieve 17 0.1 0.2 0.2
COST AND TOKENS, BY MODEL TIER
tier calls cached tokens cost usd
----------------------------------------------------
large 17 4 3903 0.018627
small 34 8 5818 0.000871
total 9721 tokens, $0.019498, mean $0.001147 per request
cache 24% hit rate, $0.005919 of calls avoided
errors 0% of requests (0% of spans, because one bad step sits next to healthy siblings)
most expensive requests: fault-03 $0.001853, warm-03 $0.001842, warm-05 $0.001832
ONE REQUEST, TRACED END TO END
request fault-04, served
trace 151aae6cb75a45bd9f86757ebde85424 total 31.4 ms spans 8
span start dur ms timeline
request 0.0 31.43 |==========================|
retrieve 0.0 0.12 |= |
rerank 0.1 0.29 |= |
llm.call 0.3 0.15 |= |
generate 0.5 30.67 |========================= |
llm.call 0.6 30.55 |========================= |
judge 31.2 0.23 | |
llm.call 31.3 0.12 | |
PRIVACY
prompts are stored as a salted hash (830d2787baae25ae, 912 characters) and not as text
spans holding raw prompt text: 0 of 136