One experiment answered that, and this page walks you through it the way it ran: scroll to watch the lattice get built, run, judged, and settled.
Seventeen real tasks, each with a pass checklist, frozen and hash-stamped before a single run. Seventeen skills, one task each.
The hash is the commitment: the tasks could not quietly change after the results started coming in.
Three Claude models at five effort levels each, plus Haiku with no effort dial, make the sixteen conditions of the grid.
One corner stays empty: Fable 5 at max never finished, it got too expensive. The slot is drawn open rather than papered over.
Each condition ran the same seventeen tasks twice, once cold and once with the skill library installed, which makes thirty-two arms.
Two blinded judges scored every checklist mark without knowing which arm they were reading, and a fixed adjudicator settled what they disagreed on.
Effort raises the floor, and the skills still add on top of it.
Bar heights are the real with-skills rates on the 17 tasks every condition completed, read from matrix.json. The three lines above quote the pre-registered low-versus-top figures from MATRIX.md; Fable's pair is low versus high, its amended confirmatory endpoint, since max never finished. Small cells are directional, not inferential.
Drag to rotate the finished lattice, or pick a cell and read its real numbers.
One experiment answered that. This is the same story the animated tour tells, as plain sections with identical facts. The full record lives on the evidence page.
Every number here is read from the published matrix · Hamza Malik · hamz.ai
Seventeen real tasks, each with a pass checklist, frozen and hash-stamped before a single run. Seventeen skills, one task each. The full slate:
The hash is the commitment: the tasks could not quietly change after the results started coming in.
Three Claude models at five effort levels each, plus Haiku with no effort dial, make the sixteen conditions of the grid. One corner stays empty: Fable 5 at max never finished, it got too expensive.
| Model | low | medium | high | xhigh (extra high) | max |
|---|---|---|---|---|---|
| Haiku 4.5 | no effort dial · one condition, ran | ||||
| Sonnet 5 | ran | ran | ran | ran | ran |
| Opus 4.8 | ran | ran | ran | ran | ran |
| Fable 5 | ran | ran | ran | ran | not run · too expensive to finish |
Each condition ran the same seventeen tasks twice, once cold and once with the skill library installed, which makes thirty-two arms.
The loaded runs get the full seventeen-skill library installed, from multi-model-adversarial-review to pre-merge-validation-gate.
Two blinded judges scored every checklist mark without knowing which arm they were reading, and a fixed adjudicator settled what they disagreed on.
Effort raises the floor, and the skills still add on top of it.
Each bar is one condition's with-skills rate; the gray base is that condition's cold rate and the green top is what the skills added. Rates use the 17 tasks every condition completed, read from matrix.json. The three lines below quote the pre-registered low-versus-top figures from MATRIX.md; Fable's pair is low versus high, its amended confirmatory endpoint, since max never finished. Small cells are directional, not inferential.
Every cell's cold rate, loaded rate, and skill delta, on the 17 tasks every condition completed.
| Model | Effort | Cold | Loaded | Skill delta | Runs |
|---|---|---|---|---|---|
| Haiku 4.5 | no dial | 64.7% | 91.2% | +26.5 points | single run per arm |
| Sonnet 5 | low | 84.8% | 96.1% | +11.3 points | mean over 3 runs per arm |
| Sonnet 5 | medium | 90.7% | 99.0% | +8.4 points | mean over 3 runs per arm |
| Sonnet 5 | high | 87.8% | 99.0% | +11.3 points | mean over 3 runs per arm |
| Sonnet 5 | xhigh | 87.3% | 98.5% | +11.3 points | mean over 3 runs per arm |
| Sonnet 5 | max | 89.7% | 99.5% | +9.8 points | mean over 3 runs per arm |
| Opus 4.8 | low | 87.3% | 95.6% | +8.3 points | mean over 3 runs per arm |
| Opus 4.8 | medium | 85.8% | 97.6% | +11.8 points | mean over 3 runs per arm |
| Opus 4.8 | high | 88.2% | 98.0% | +9.8 points | mean over 3 runs per arm |
| Opus 4.8 | xhigh | 91.2% | 99.0% | +7.8 points | mean over 3 runs per arm |
| Opus 4.8 | max | 91.2% | 100.0% | +8.8 points | mean over 3 runs per arm |
| Fable 5 | low | 94.1% | 99.0% | +4.9 points | mean over 3 runs per arm |
| Fable 5 | medium | 92.6% | 100.0% | +7.4 points | single run per arm |
| Fable 5 | high | 97.6% | 100.0% | +2.4 points | mean over 3 runs per arm |
| Fable 5 | xhigh | 97.1% | 100.0% | +2.9 points | single run per arm |
| Fable 5 | max | not run · it got too expensive to finish | |||
The full record lives on the evidence page. The harness, tasks, and judge inputs are in the repo.