Type a customer name and see which version of a prompt they get, and whether the new one earns its promotion.
When a team changes the wording of a prompt, two questions follow. Which customers are seeing the new version, and is it actually better? Both are easy to get wrong: a split that reshuffles on every request means one customer sees both versions and the comparison measures nothing.
Type a customer or company name. The page works out which of the two versions that name gets, shows that the answer is the same every time it is asked, and shows the same name landing somewhere else in a different experiment. There is no table of assignments anywhere. The arm is worked out from the name itself, so it survives a restart and can be recalculated later from a log.
Then six hundred customers go through both versions for real, each one answering a question with a known correct answer, and the results are put in front of a gate that decides whether the new version ships. The page shows the gate refusing on a small sample even though the numbers already look convincing, and agreeing later on the full sample, with the odds that the difference is chance.
THE TWO VERSIONS OF THIS PROMPT
control ve0c6cbe24707 one supporting document, answer it
challenger v324d23676128 three tagged documents, quote and cite the one you used
the id is a hash of the wording and the settings, so an edit cannot keep the old id
live right now: prod -> ve0c6cbe24707, staging -> v324d23676128
==============================================================================
WHICH VERSION DOES "meridian-labs" GET
==============================================================================
the name is hashed with the experiment name into a number from 0 to 9999
"meridian-labs" lands on 9364, which is 93.6% of the way along the line
the line is split 50/50, so this name gets: CHALLENGER (v324d23676128)
asked five times in a row: challenger
asked again from a splitter built from scratch: challenger (a restart changes nothing)
the same name in a different experiment: control (a different arm)
no assignment is written down anywhere. the answer is recalculated from the name,
which is why an analyst can rebuild who saw what months later from a log alone.
==============================================================================
600 CUSTOMERS PUT THROUGH BOTH VERSIONS FOR REAL
==============================================================================
each one asks a question with a known correct answer, and the answer either
contains the fact it should or it does not. nothing here is a random number.
your row: "meridian-labs" asked "How is a failed invoice payment retried?"
served the challenger version, correct answer present: no (343 tokens)
arm version samples success rate p50 ms p95 ms tokens cost usd
-----------------------------------------------------------------------------------------
challenger v324d23676128 297 136 45.8% 0.18 0.20 326.2 0.0209
control ve0c6cbe24707 303 97 32.0% 0.09 0.10 136.3 0.0095
traffic split: control 50.5%, challenger 49.5% against an intended 50/50, largest gap 0.50 points
a hash split is never exactly even, so the thing to watch is the size of the gap
==============================================================================
SHOULD THE NEW VERSION SHIP
==============================================================================
AFTER THE FIRST 40 REQUESTS
verdict: HOLD
control control 6/21 = 28.6% 95% CI [13.8%, 50.0%]
challenger challenger 11/19 = 57.9% 95% CI [36.3%, 76.9%]
absolute lift +29.3 points, z = 1.873, p = 0.06101
- not enough data: control has 21 and challenger has 19, the gate requires 200 per arm
the odds this gap is chance are 6.1% here, but the gate never got
as far as looking: the sample floor is checked first
for an effect this size the sample really needed is about 1344 per version
this checkpoint shows a lift of +29.3 points. the full sample below settles at +13.8.
the floor is a promise made before the numbers arrived, which is the only reason
the decision is still open at this point.
AFTER ALL 600 REQUESTS
verdict: PROMOTE
control control 97/303 = 32.0% 95% CI [27.0%, 37.5%]
challenger challenger 136/297 = 45.8% 95% CI [40.2%, 51.5%]
absolute lift +13.8 points, z = 3.462, p = 0.00054
- challenger beats control by 13.8 points (p = 0.00054) on 303 and 297 samples
enough data: yes, 303 and 297 against a floor of 200 each
odds this gap is chance: 0.054%
THE SAME RESULT UNDER A COST CEILING
the new version sends 2.39x the words and costs 2.20x as much
verdict: HOLD
control control 97/303 = 32.0% 95% CI [27.0%, 37.5%]
challenger challenger 136/297 = 45.8% 95% CI [40.2%, 51.5%]
absolute lift +13.8 points, z = 3.462, p = 0.00054
- cost ratio 2.20 exceeds the 1.50 ceiling
==============================================================================
WHAT HAPPENS NEXT
==============================================================================
promoted: prod now points at v324d23676128
rolled back in one call: prod is ve0c6cbe24707 again
nothing was copied between environments. a label is a pointer, and rollback moves
it back to whatever the written-down history says it pointed at before.
saved and reloaded: 2 versions, 5 recorded changes, prod pointer intact: True
editing that file by hand breaks the id it claims, and loading it fails on purpose
who changed what:
17:01:52 support_answer/staging (unset) -> v324d23676128 by arnob promote dev to staging
17:01:52 support_answer/prod (unset) -> ve0c6cbe24707 by arnob initial production prompt
17:01:53 support_answer/prod ve0c6cbe24707 -> v324d23676128 by ci-bot gate passed, p=0.00054
17:01:53 support_answer/prod v324d23676128 -> ve0c6cbe24707 by oncall cost ceiling flagged in review