Eight questions a tester typed. The prompt never saw them. The headline is pass^5 per experiment. The interval on pass^10 says how sure to be. The pooled rate is for comparing prompts during the climb. The dev set was never run at xhigh.
Each row is two experiments of five runs on the same prompt. The highlighted row is what ships.
| Milestone | Set | Judge | Effort · cap | pass^5 each | pass^10 · Wilson 95% | Pooled pass^1 | Cut at cap | Cost |
|---|
Rows through the corrected rerun used the old judge. The sweep used the xhigh judge, so 65 to 69 is a judge change. Prompt hash b44a9181 on every row. Dev was never run at xhigh.
Rows grouped by tier. Ten columns, two experiments. A solid row is in pass^10.
Scorers run in table order. The letter is the first that failed. Hover a cell.
Dev experiments only. A label marks a prompt change. Each prompt ran twice before it was judged, because the same prompt moves between runs.
Counts are for this pair only.
Every run in the headline pair, seconds from question to answer. Most of it is the model thinking. 500 s is the harness cap.
Latency is comparable only between passes with the same worker count.