Pick a version of the answering system, watch it get graded on 23 questions, and see whether the merge gate lets it through.
Type healthy, noisy or degraded to choose which version of the question answering system gets tested. All three are real code. The healthy one reads two sentences of evidence per question and refuses to answer anything the documents do not cover. The noisy one reads three sentences instead of two. The degraded one never refuses and cuts every answer down to six words.
The page grades that version against 23 questions, shows you the result of every single one, and then gives you the average with a confidence interval, so you can see how much of the number is real and how much is the small sample size.
Then it runs the check that would run on a pull request. The gate compares this run against a stored baseline and blocks only when the drop is bigger than the allowed limit and the evidence actually supports calling it a regression. A drop that could just be noise is reported as a warning instead of blocking. The exit code printed at the bottom is the one the build would return.
SYSTEM UNDER TEST version degraded config refusal switched off, every answer clipped to 6 words dataset 23 cases (15 from the docs, 8 written to be hard) scoring deterministic string checks only, the same ones the baseline was measured with EVERY CASE, ONE ROW EACH case difficulty result scores -------------------------------------------------------------------------- gold-01 easy pass contains 1.00, token_f1 0.92 gold-02 easy FAIL contains 0.00, token_f1 0.15 gold-03 medium FAIL contains 0.00, token_f1 0.00 gold-04 easy FAIL contains 0.00, token_f1 0.00 gold-05 easy FAIL contains 0.00, token_f1 0.52 gold-06 medium FAIL contains 0.00, token_f1 0.00 gold-07 easy FAIL contains 0.00, token_f1 0.00 gold-08 easy FAIL contains 0.00, token_f1 0.13 gold-09 medium FAIL contains 0.00, token_f1 0.50 gold-10 easy FAIL contains 0.00, token_f1 0.08 gold-11 easy FAIL contains 0.00, token_f1 0.14 gold-12 medium FAIL contains 0.00, token_f1 0.00 gold-13 easy FAIL contains 0.00, token_f1 0.18 gold-14 easy FAIL contains 0.00, token_f1 0.13 gold-15 medium FAIL contains 0.00, token_f1 0.00 adv-01 hard FAIL regex 0.00 adv-02 hard FAIL contains 0.00 adv-03 hard FAIL contains 0.00 adv-04 medium FAIL contains 0.00, token_f1 0.13 adv-05 medium pass json_schema 1.00 adv-06 hard pass contains 1.00 adv-07 medium FAIL contains 0.00 adv-08 hard pass regex 1.00 4 of 23 cases passed every scorer they asked for AGGREGATE, WITH A BOOTSTRAP CONFIDENCE INTERVAL (2000 resamples, fixed seed, so this page is reproducible) metric mean 95% CI cases --------------------------------------------------- contains 0.100 [0.000, 0.250] 20 json_schema 1.000 [1.000, 1.000] 1 pass_rate 0.174 [0.043, 0.348] 23 regex 0.500 [0.000, 1.000] 2 token_f1 0.180 [0.075, 0.317] 16 slice (3+ cases only) pass rate cases ----------------------------------------- difficulty:easy 0.100 10 difficulty:hard 0.400 5 difficulty:medium 0.125 8 tag:adversarial 0.375 8 tag:gold 0.067 15 tag:grounded 0.067 15 4 of 23 cases were settled by free string checks, so an LLM judge running only on failures would cost 19 calls instead of 23. A COUPLE OF THE ACTUAL ANSWERS passed gold-01: Meridian tokens expire 90 days after ... failed gold-02: The default rate limit is 600 ... THE MERGE GATE, AGAINST THE COMMITTED BASELINE baseline file: projects/p04_eval_harness/data/baseline.json metric baseline this run change limit verdict -------------------------------------------------------------------------- contains 0.700 0.100 -0.600 0.020 regression json_schema 1.000 1.000 +0.000 0.020 ok pass_rate 0.696 0.174 -0.522 0.020 regression regex 1.000 0.500 -0.500 0.020 noise token_f1 0.494 0.180 -0.314 0.020 regression VERDICT: BLOCKED. 3 metric(s) breached. blocked on contains: dropped 0.600 (limit 0.020); baseline 0.700 is outside the candidate interval [0.000, 0.250] blocked on pass_rate: dropped 0.522 (limit 0.020); baseline 0.696 is outside the candidate interval [0.043, 0.348] blocked on token_f1: dropped 0.314 (limit 0.020); baseline 0.494 is outside the candidate interval [0.075, 0.317] warning: regex dropped 0.500 but the baseline sits inside the candidate interval [0.000, 1.000]; n=2 is too small to call this a regression the build would exit with code 1 (merge blocked) a drop only blocks when it is bigger than the limit AND the baseline sits outside this run's confidence interval; anything else is a warning, because a gate that fires on noise gets switched off