Pick a version of the answering system, watch it get graded on 23 questions, and see whether the merge gate lets it through.
Type healthy, noisy or degraded to choose which version of the question answering system gets tested. All three are real code. The healthy one reads two sentences of evidence per question and refuses to answer anything the documents do not cover. The noisy one reads three sentences instead of two. The degraded one never refuses and cuts every answer down to six words.
The page grades that version against 23 questions, shows you the result of every single one, and then gives you the average with a confidence interval, so you can see how much of the number is real and how much is the small sample size.
Then it runs the check that would run on a pull request. The gate compares this run against a stored baseline and blocks only when the drop is bigger than the allowed limit and the evidence actually supports calling it a regression. A drop that could just be noise is reported as a warning instead of blocking. The exit code printed at the bottom is the one the build would return.
SYSTEM UNDER TEST version healthy config k=2 evidence sentences, refuses when the docs do not cover it dataset 23 cases (15 from the docs, 8 written to be hard) scoring deterministic string checks only, the same ones the baseline was measured with EVERY CASE, ONE ROW EACH case difficulty result scores -------------------------------------------------------------------------- gold-01 easy pass contains 1.00, token_f1 0.58 gold-02 easy FAIL contains 0.00, token_f1 0.11 gold-03 medium pass contains 1.00, token_f1 0.76 gold-04 easy FAIL contains 0.00, token_f1 0.12 gold-05 easy pass contains 1.00, token_f1 0.97 gold-06 medium FAIL contains 0.00, token_f1 0.00 gold-07 easy FAIL contains 1.00, token_f1 0.00 gold-08 easy pass contains 1.00, token_f1 0.76 gold-09 medium pass contains 1.00, token_f1 0.71 gold-10 easy pass contains 1.00, token_f1 0.82 gold-11 easy pass contains 1.00, token_f1 0.80 gold-12 medium FAIL contains 0.00, token_f1 0.08 gold-13 easy pass contains 1.00, token_f1 0.57 gold-14 easy pass contains 1.00, token_f1 0.69 gold-15 medium FAIL contains 0.00, token_f1 0.17 adv-01 hard pass regex 1.00 adv-02 hard FAIL contains 0.00 adv-03 hard pass contains 1.00 adv-04 medium pass contains 1.00, token_f1 0.76 adv-05 medium pass json_schema 1.00 adv-06 hard pass contains 1.00 adv-07 medium pass contains 1.00 adv-08 hard pass regex 1.00 16 of 23 cases passed every scorer they asked for AGGREGATE, WITH A BOOTSTRAP CONFIDENCE INTERVAL (2000 resamples, fixed seed, so this page is reproducible) metric mean 95% CI cases --------------------------------------------------- contains 0.700 [0.500, 0.900] 20 json_schema 1.000 [1.000, 1.000] 1 pass_rate 0.696 [0.522, 0.870] 23 regex 1.000 [1.000, 1.000] 2 token_f1 0.494 [0.328, 0.654] 16 slice (3+ cases only) pass rate cases ----------------------------------------- difficulty:easy 0.700 10 difficulty:hard 0.800 5 difficulty:medium 0.625 8 tag:adversarial 0.875 8 tag:gold 0.600 15 tag:grounded 0.600 15 16 of 23 cases were settled by free string checks, so an LLM judge running only on failures would cost 7 calls instead of 23. A COUPLE OF THE ACTUAL ANSWERS passed gold-01: Meridian tokens expire 90 days after creation [S2]. Meridian authenticates every request with a ... failed gold-02: The default rate limit is 600 requests per minute per workspace, burstable to 1000 for 10 second... THE MERGE GATE, AGAINST THE COMMITTED BASELINE baseline file: projects/p04_eval_harness/data/baseline.json metric baseline this run change limit verdict -------------------------------------------------------------------------- contains 0.700 0.700 +0.000 0.020 ok json_schema 1.000 1.000 +0.000 0.020 ok pass_rate 0.696 0.696 +0.000 0.020 ok regex 1.000 1.000 +0.000 0.020 ok token_f1 0.494 0.494 +0.000 0.020 ok VERDICT: ALLOWED. No metric dropped at all, so the 0.020 limit was never approached. the build would exit with code 0 (merge allowed) a drop only blocks when it is bigger than the limit AND the baseline sits outside this run's confidence interval; anything else is a warning, because a gate that fires on noise gets switched off