Prompt Lab review: useful comparison rules, scorer fixes still needed
In brief
Prompt Lab gives prompt A/B testing a shared fixture set and declared rubric. A source review and small local scorer checks found implementation limits that need fixing before its totals can support a decision.
Originally published on Jack Gross.
Prompt Lab gives prompt A/B testing a shared fixture set and declared rubric. A source review and small local scorer checks found implementation limits that need fixing before its totals can support a decision.
What is being reviewed
This review covers public version0.1.1, revision e7e5027c6e6522bf8e2b6dae9ff25744c9451f35, checked October29,2026. Prompt Lab is a ZCode plugin with a Python standard-library scorer. Its MIT license permits reuse subject to the license terms. I inspected the README, skill workflow and scorer, then ran tiny synthetic scoring cases locally. I did not run ZCode, spawn model comparisons or measure provider costs. The project was submitted for review by its owner; the findings below are independent of the referral links on this site.
The useful design
Both variants receive the same fixtures, with a weighted rubric declared before judging. The documented workflow asks for a fresh agent for each fixture/variant/run and records runs.json beside the outputs. It keeps the rubric and expected properties out of those execution prompts. That is a sensible way to reduce cross-variant leakage. The scorer itself does not create agents or verify that isolation happened; a log records a claim about execution, not proof of a clean context. The README documents marketplace installation and /prompt-lab with two prompt paths, fixtures, rubric and a run count.
Two numerical checks that failed
For one fixture I wrote A-run1.txt containing OK and A-run2.txt containing FAIL, with a contains-OK criterion. At runs_per=2 the scorer read OK for both runs and returned1.0. In compute_scores, the inner read calls collect_outputs with1 each time, rather than selecting the current run. A separate two-fixture check, first passing and second failing, reported fixture scores1.0 and0.5; the second should be0.0 for that criterion alone. The got_w and total_w accumulators span earlier fixtures, so per-fixture scores are cumulative. These checks test arithmetic and file selection, not a model.
Pending judgment and literal checks
With one objective criterion and one unjudged manual criterion, aggregation still printed a winner while both variants had pending_manual=1. The pending count is visible, which helps, but it does not enforce the README promise of a verdict withheld until the manual work is complete. Also, the expected property no invented numbers becomes a literal absence check for the phrase invented numbers. It cannot detect a made-up value. Contains and regex are useful for format constraints; factual correctness and causal claims about which prompt wording mattered need separate judgment.
What I would change before trusting a total
Read the exact run number, fail closed when that output is absent, and reset score accumulators at each run and fixture. Block a final verdict while required manual judgments remain, or label it provisional. Treat semantic properties as manual and validate the audit trail against the requested run matrix. Regression tests should include different run outputs, a missing second run, pass/fail fixture order reversal and pending-manual cases. These are recommendations, not fixes applied to the repository. Until they pass, I would use the workflow as a checklist and inspect raw outputs instead of using the current totals to pick a prompt.