Pick a version of the answering system, watch it get graded on 23 questions, and see whether the merge gate lets it through.
Type healthy, noisy or degraded to choose which version of the question answering system gets tested. All three are real code. The healthy one reads two sentences of evidence per question and refuses to answer anything the documents do not cover. The noisy one reads three sentences instead of two. The degraded one never refuses and cuts every answer down to six words.
The page grades that version against 23 questions, shows you the result of every single one, and then gives you the average with a confidence interval, so you can see how much of the number is real and how much is the small sample size.
Then it runs the check that would run on a pull request. The gate compares this run against a stored baseline and blocks only when the drop is bigger than the allowed limit and the evidence actually supports calling it a regression. A drop that could just be noise is reported as a warning instead of blocking. The exit code printed at the bottom is the one the build would return.
SYSTEM UNDER TEST version noisy config k=3 evidence sentences, one retrieval knob moved dataset 23 cases (15 from the docs, 8 written to be hard) scoring deterministic string checks only, the same ones the baseline was measured with EVERY CASE, ONE ROW EACH case difficulty result scores -------------------------------------------------------------------------- gold-01 easy FAIL contains 0.00, token_f1 0.15 gold-02 easy FAIL contains 0.00, token_f1 0.17 gold-03 medium pass contains 1.00, token_f1 0.68 gold-04 easy FAIL contains 0.00, token_f1 0.00 gold-05 easy pass contains 1.00, token_f1 0.74 gold-06 medium FAIL contains 0.00, token_f1 0.00 gold-07 easy FAIL contains 1.00, token_f1 0.00 gold-08 easy pass contains 1.00, token_f1 0.69 gold-09 medium FAIL contains 1.00, token_f1 0.28 gold-10 easy pass contains 1.00, token_f1 0.83 gold-11 easy pass contains 1.00, token_f1 0.89 gold-12 medium FAIL contains 0.00, token_f1 0.07 gold-13 easy FAIL contains 0.00, token_f1 0.05 gold-14 easy pass contains 1.00, token_f1 0.69 gold-15 medium pass contains 1.00, token_f1 0.52 adv-01 hard pass regex 1.00 adv-02 hard pass contains 1.00 adv-03 hard FAIL contains 0.00 adv-04 medium pass contains 1.00, token_f1 0.76 adv-05 medium pass json_schema 1.00 adv-06 hard pass contains 1.00 adv-07 medium pass contains 1.00 adv-08 hard pass regex 1.00 14 of 23 cases passed every scorer they asked for AGGREGATE, WITH A BOOTSTRAP CONFIDENCE INTERVAL (2000 resamples, fixed seed, so this page is reproducible) metric mean 95% CI cases --------------------------------------------------- contains 0.650 [0.450, 0.850] 20 json_schema 1.000 [1.000, 1.000] 1 pass_rate 0.609 [0.391, 0.783] 23 regex 1.000 [1.000, 1.000] 2 token_f1 0.407 [0.245, 0.574] 16 slice (3+ cases only) pass rate cases ----------------------------------------- difficulty:easy 0.500 10 difficulty:hard 0.800 5 difficulty:medium 0.625 8 tag:adversarial 0.875 8 tag:gold 0.467 15 tag:grounded 0.467 15 14 of 23 cases were settled by free string checks, so an LLM judge running only on failures would cost 9 calls instead of 23. A COUPLE OF THE ACTUAL ANSWERS passed gold-03: Only owners can delete a workspace, export the audit log or change billing [S3]. A workspace is ... failed gold-01: A failed payment retries on days 3, 7 and 14 before the workspace is suspended [S3]. Meridian au... THE MERGE GATE, AGAINST THE COMMITTED BASELINE baseline file: projects/p04_eval_harness/data/baseline.json metric baseline this run change limit verdict -------------------------------------------------------------------------- contains 0.700 0.650 -0.050 0.020 noise json_schema 1.000 1.000 +0.000 0.020 ok pass_rate 0.696 0.609 -0.087 0.020 noise regex 1.000 1.000 +0.000 0.020 ok token_f1 0.494 0.407 -0.087 0.020 noise VERDICT: ALLOWED. The largest drop was pass_rate at -0.087 against a limit of 0.020. warning: contains dropped 0.050 but the baseline sits inside the candidate interval [0.450, 0.850]; n=20 is too small to call this a regression warning: pass_rate dropped 0.087 but the baseline sits inside the candidate interval [0.391, 0.783]; n=23 is too small to call this a regression warning: token_f1 dropped 0.087 but the baseline sits inside the candidate interval [0.245, 0.574]; n=16 is too small to call this a regression the build would exit with code 0 (merge allowed) a drop only blocks when it is bigger than the limit AND the baseline sits outside this run's confidence interval; anything else is a warning, because a gate that fires on noise gets switched off