Every number below is read from the committed result files at the moment this page was built; the template only supplies labels. Every box is a small sample, so read each rate and change as a rough signal, not a precise measurement.
results/matrix/matrix.json, 17 run directories matching results/lattice-*,
and results/concordance/codex-concordance.jsonpython3 eval/make_explorer.py --matrix results/matrix/matrix.json --cells 'results/lattice-*' --out results/matrix/explorer.html
Each box is one model run at one effort level, scored two ways: without the skill installed and with it. Each task carries a short checklist of things a correct answer must do, and a score is the share of those checklist items hit. Change is the with-skill score minus the without-skill score, in percentage points. The rules and pass criteria were locked and publicly hash-stamped before the first run. Every box is a small sample, so read each number as a rough signal, not a precise measurement.
Codex 5.6, one run, 50 sampled comparisons, exploratory: agreed 97.2%. A spot-check by a second vendor, not a re-run of the experiment. Turn it on to mark the boxes whose comparisons Codex re-scored.
| model \ effort | low | medium | high | xhigh | max | no effort(reference) |
|---|---|---|---|---|---|---|
| claude-fable-5 | ×3 runswithout 94.1%with 99.0%change +4.917 tasks, 3 runs eachCodex 29/30 agree | without 92.6%with 100.0%change +7.417 tasks, 1 run eachCodex 24/26 agree | ×3 runswithout 97.6%with 100.0%change +2.417 tasks, 3 runs eachCodex 56/58 agree | without 97.1%with 100.0%change +2.917 tasks, 1 run eachCodex 14/14 agree | max run not finishedit got too expensive to finish; one set of runs is still going, so this box may fill in later | - |
| claude-sonnet-5 | ×3 runswithout 84.8%with 96.1%change +11.317 tasks, 3 runs eachCodex 32/32 agree | ×3 runswithout 90.7%with 99.0%change +8.417 tasks, 3 runs each | ×3 runswithout 87.8%with 99.0%change +11.317 tasks, 3 runs each | ×3 runswithout 87.3%with 98.5%change +11.317 tasks, 3 runs each | ×3 runswithout 89.7%with 99.5%change +9.817 tasks, 3 runs eachCodex 40/40 agree | - |
| claude-opus-4-8 | ×3 runswithout 87.3%with 95.6%change +8.317 tasks, 3 runs eachCodex 51/52 agree | ×3 runswithout 85.8%with 97.6%change +11.817 tasks, 3 runs each | ×3 runswithout 88.2%with 98.0%change +9.817 tasks, 3 runs each | ×3 runswithout 91.2%with 99.0%change +7.817 tasks, 3 runs each | ×3 runswithout 91.2%with 100.0%change +8.817 tasks, 3 runs eachCodex 36/38 agree | - |
| claude-haiku-4-5-20251001no effort setting, shown for reference | - | - | - | - | - | without 64.7%with 91.2%change +26.517 tasks, 1 run eachCodex 16/16 agree |
The same checklist scores drawn against effort. Each model has its own colour. The solid line is with the skill installed, the dashed line is without. Low and max were run three times; the middle levels once.
claude-haiku-4-5-20251001 has no effort setting, so it sits outside the effort comparison: without the skill 64.7%, with the skill 91.2%, change +26.5 (17 tasks, 1 run each).
per-skill PASS/FAIL verdicts are deliberately suppressed in lattice outputs and subordinated to the per-cell run reports; skills appear here as rates only Each cell below shows checklist hits for that skill's one task within each condition. Every per-skill cell is a single task, so these are the roughest signals on the page.
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 2/366.7% | 3/3100.0% | +33.3points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 1/333.3% | 2/366.7% | +33.3points | 1 task, 1 run each |
| claude-opus-4-8@low | 2/366.7% | 2/366.7% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 2/366.7% | 3/3100.0% | +33.3points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 2/366.7% | 2/366.7% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 2/450.0% | 4/4100.0% | +50.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 2/450.0% | 4/4100.0% | +50.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 2/450.0% | 4/4100.0% | +50.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/580.0% | 4/580.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/560.0% | 5/5100.0% | +40.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 0/40.0% | 2/450.0% | +50.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 3/475.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 3/475.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 2/450.0% | 4/4100.0% | +50.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 2/450.0% | 3/475.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 3/475.0% | -25.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 3/475.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/3100.0% | 3/3100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 2/450.0% | 4/4100.0% | +50.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 4/4100.0% | 3/475.0% | -25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-fable-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 1 run each |
| claude-fable-5@high | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 3/560.0% | 4/580.0% | +20.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 4/580.0% | 5/5100.0% | +20.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 5/5100.0% | 5/5100.0% | +0.0points | 1 task, 3 runs each |
| condition | without skill | with skill | change | sample |
|---|---|---|---|---|
| claude-fable-5@low | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 1 run each |
| claude-fable-5@high | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-fable-5@xhigh | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 1 run each |
| claude-haiku-4-5-20251001@none | 1/425.0% | 4/4100.0% | +75.0points | 1 task, 1 run each |
| claude-opus-4-8@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-opus-4-8@max | 4/4100.0% | 4/4100.0% | +0.0points | 1 task, 3 runs each |
| claude-sonnet-5@low | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@medium | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@high | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@xhigh | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
| claude-sonnet-5@max | 3/475.0% | 4/4100.0% | +25.0points | 1 task, 3 runs each |
Every blinded comparison was scored by two independent judges, one Sonnet-class and one Opus-class. Where they split on a checklist mark, a pinned third judge, Claude Fable 5, decided it, and the final mark is the majority of the three. Disputed marks stay in every total.
| condition | judges split | sent to third judge | unresolved |
|---|---|---|---|
| claude-fable-5@low | 9 / 2044.4% | 10 / 4082.5% | 0 |
| claude-fable-5@medium | 1 / 681.5% | 1 / 1360.7% | 0 |
| claude-fable-5@high | 9 / 2044.4% | 10 / 4082.5% | 0 |
| claude-fable-5@xhigh | 3 / 684.4% | 3 / 1362.2% | 0 |
| claude-haiku-4-5-20251001@none | 9 / 6813.2% | 10 / 1367.4% | 0 |
| claude-opus-4-8@low | 14 / 2046.9% | 15 / 4083.7% | 0 |
| claude-opus-4-8@medium | 9 / 2044.4% | 11 / 4082.7% | 0 |
| claude-opus-4-8@high | 9 / 2044.4% | 11 / 4082.7% | 0 |
| claude-opus-4-8@xhigh | 6 / 2042.9% | 7 / 4081.7% | 0 |
| claude-opus-4-8@max | 8 / 2043.9% | 9 / 4082.2% | 0 |
| claude-sonnet-5@low | 8 / 2043.9% | 10 / 4082.5% | 0 |
| claude-sonnet-5@medium | 9 / 2044.4% | 9 / 4082.2% | 0 |
| claude-sonnet-5@high | 9 / 2044.4% | 9 / 4082.2% | 0 |
| claude-sonnet-5@xhigh | 5 / 2042.5% | 5 / 4081.2% | 0 |
| claude-sonnet-5@max | 14 / 2046.9% | 15 / 4083.7% | 0 |
Codex 5.6, one run, 50 sampled comparisons, exploratory: agreed 97.2%
A different vendor's model, Codex 5.6 (gpt-5.6-terra), re-scored 50 of the comparisons using the same frozen instructions the Claude judges saw. This is a confidence spot-check, not a re-run of the experiment, and it changes none of the numbers above. Use the toggle in section 1 to see which boxes these comparisons came from.
| agreement by model column | Fable | Sonnet | Opus | Haiku |
|---|---|---|---|---|
| Codex vs final marks | 96.1%123/128 marks | 97.5%119/122 marks | 97.7%125/128 marks | 100.0%16/16 marks |