System 04

LLM Evaluation Harness with a CI Quality Gate

Pick a version of the answering system, watch it get graded on 23 questions, and see whether the merge gate lets it through.

What it does

Type healthy, noisy or degraded to choose which version of the question answering system gets tested. All three are real code. The healthy one reads two sentences of evidence per question and refuses to answer anything the documents do not cover. The noisy one reads three sentences instead of two. The degraded one never refuses and cuts every answer down to six words.

The page grades that version against 23 questions, shows you the result of every single one, and then gives you the average with a confidence interval, so you can see how much of the number is real and how much is the small sample size.

Then it runs the check that would run on a pull request. The gate compares this run against a stored baseline and blocks only when the drop is bigger than the allowed limit and the evidence actually supports calling it a regression. A drop that could just be noise is reported as a warning instead of blocking. The exit code printed at the bottom is the one the build would return.

Try it

healthynoisydegraded