Both levers work. They do different jobs.
More reasoning effort made every model better on its own. Comparing the lowest effort setting against the highest (high on Fable, whose max runs never finished), scores without any skills installed rose 3.5 percentage points on Fable, 3.9 on Opus, and 4.9 on Sonnet. Each cleared the 3-point bar fixed before the first run. More thinking buys better work, reliably.
It did not buy back what the skills add. Runs with the skills installed beat runs without them at every effort level, on every model. At the highest effort settings the skills still added between 2.4 and 9.8 points, and on Opus the gap at max effort came out a touch wider than at low, the one model where the skills ended up adding more at the top of the dial than at the bottom. If more thinking could substitute for packaged expertise, that advantage should have shrunk toward zero, and it never came close. On Haiku, the smallest model here and the one with no effort dial at all, the skills added 26 points, the largest gap in the study. That Haiku figure is a single run per side and sits outside the effort trend, so read it as a separate result, not another point on the curve.
There is a headroom story in the grid too. The skills helped every model, and they helped Fable the least, because Fable starts from the highest floor. With the library installed, Fable never scored below 99.0. Opus reached 100.0 at max effort. Sonnet, which starts cold in the 80s, reached 99.5 at max. Less headroom means less to add, not nothing to add.
Two terms before the numbers. Each task carries a short checklist of things a correct answer must do, and a score is the share of checklist items hit. Blinded means the two Claude judges marking each item were never told which answer ran with the skills installed; a fixed third judge settled any disagreement.
Share of checklist items hit, on the 17 tasks every setup completed. The highlighted column is what installing the skills added, in percentage points. Most cells were run three times per side for stability; the single runs are Fable at medium and extra-high and the Haiku baseline.
| Model | Effort | Without skills | With skills | What the skills added |
|---|---|---|---|---|
| Haiku 4.5 | no effort dialruns one way | 64.7%1 run | 91.2%1 run | +26.5points |
| Sonnet 5 | low | 84.8%avg of 3 runs | 96.1%avg of 3 runs | +11.3points |
| medium | 90.7%avg of 3 runs | 99.0%avg of 3 runs | +8.4points | |
| high | 87.8%avg of 3 runs | 99.0%avg of 3 runs | +11.3points | |
| xhigh (extra high) | 87.3%avg of 3 runs | 98.5%avg of 3 runs | +11.3points | |
| max | 89.7%avg of 3 runs | 99.5%avg of 3 runs | +9.8points | |
| Opus 4.8 | low | 87.3%avg of 3 runs | 95.6%avg of 3 runs | +8.3points |
| medium | 85.8%avg of 3 runs | 97.6%avg of 3 runs | +11.8points | |
| high | 88.2%avg of 3 runs | 98.0%avg of 3 runs | +9.8points | |
| xhigh | 91.2%avg of 3 runs | 99.0%avg of 3 runs | +7.8points | |
| max | 91.2%avg of 3 runs | 100.0%avg of 3 runs | +8.8points | |
| Fable 5 | low | 94.1%avg of 3 runs | 99.0%avg of 3 runs | +4.9points |
| medium | 92.6%1 run | 100.0%1 run | +7.4points | |
| high | 97.6%avg of 3 runs | 100.0%avg of 3 runs | +2.4points | |
| xhigh | 97.1%1 run | 100.0%1 run | +2.9points | |
| max | did not finish · it got too expensiveslot open · one run is still quietly going | |||
Methodology: each score is the share of checklist items hit across all 17 tasks, scored by two blinded Claude judges with a third settling disagreements. Every setup completed every task, so all 15 cells are scored on the same 17-task set. Rows marked avg of 3 runs average three runs per side; rows marked 1 run are single runs, so on those read the direction, not the decimals. Source: MATRIX.md in the repo.
One control for both charts below. “With skills” keeps the three solid lines. “Effort effect” keeps the three lines without skills, which is what the effort dial does on its own.
Share of checklist items hit (%), by effort level. Full 0 to 100 scale.
Methodology: share of checklist items hit on the 17 tasks every setup completed, scored by two blinded Claude judges with a third settling disagreements. Each point sits on its labeled effort column; most points average three runs per side, with Fable at medium and extra-high and the Haiku baseline the single runs. Fable's lines stop at xhigh because its max runs got too expensive to finish. Source: MATRIX.md in the repo.
Same data as above, y-axis zoomed to 80 to 100 to show the differences.
Methodology: identical data, sources, and run counts to the chart above, y-axis windowed to 80 to 100. Haiku is omitted here because its score without the skills, 64.7%, sits below the window.
Checklist score against the average recorded cost of one run, USD. Each point is one model at one effort level.
One model at a time keeps the chart readable: solid is with the skills installed, dashed is without. “All models” shows the with-skills lines side by side; hover any point for its numbers.
Methodology: the horizontal axis is each cell's average recorded cost of one run, on a log scale; the vertical axis is the share of checklist items hit on the shared 17 tasks. Costs are observed spend per run on subscription pricing, not list-price API rates, and the run counts behind each average are small: 51 runs per side on the replicated cells (three repeats of the 17 tasks), and 17 on Fable's medium and extra-high settings and the Haiku baseline, the study's only single runs. Turning effort up multiplied per-run cost by up to 2.7x. Loading the skills cost between $0.01 and $0.30 per run, and on Sonnet's two highest settings the runs with the skills installed averaged slightly cheaper than the runs without. Sources: cost-per-run-dataset.json, published next to this page, and MATRIX.md in the repo.
Every judge in this study is a Claude model scoring Claude outputs, which is a fair thing to be skeptical of. As a check, Codex 5.6, a model from a different vendor, re-scored 50 sampled comparisons in a single run and agreed with the final marks 97.2% of the time. Full detail in the full methodology.
All in, the experiment cost $1,296.94: $1,034.05 running the arms and $262.89 judging them, with the cost of building the harness excluded. Every run recorded its own cost as it ran, so these are sums of real records, not estimates.
Effort is the expensive dial. Going from low to max multiplied the cost of a run by up to 2.7x; a Sonnet run went from about $0.44 to about $1.20. Loading the skills cost between $0.01 and $0.30 per run.
At Sonnet's two highest settings, runs with the skills installed actually averaged slightly cheaper than runs without. The exact per-cell numbers are in the cost table.
What this means for how you use AI
Pick the model and effort for the floor you need. Effort reliably raises the baseline, and you pay for it on every run. Buy as much of it as the job actually requires, and no more.
Install skills for the failure modes that matter to you. They kept paying at every effort level, on every model, for cents per run. A skill is built once and rides along for free on every run after that.
Measure both instead of guessing. The two levers moved separately here, and nothing about your tasks is settled by someone else's benchmark. The whole instrument is open if you want to run it on your own work.
How the grid, the judging, and the costs worked is in How it worked. Every rule, bug, and disclosure is in The full methodology.
How it worked
Seventeen engineering tasks, a library of seventeen skills, and sixteen model-and-effort setups. Every setup answered every task twice, once with the skill library installed and once without, and blinded judges scored all of it.
What was tested: the tasks
The tasks are realistic, hands-on engineering scenarios, each set in a small purpose-built codebase. The suite of 17 was frozen before the first run. Three examples, straight from the suite:
- Review a teammate's pull request that adds appointment search to a multi-tenant clinic scheduler, and decide whether it is safe to merge.
- Work out why a nightly ticket-classifying job's accuracy fell from about 0.91 to zero and stayed there, even though individual predictions still look fine when spot-checked.
- Audit a pipeline that grades customer-support replies on three axes and ranks agents by their combined score.
What was tested: the skill library
A skill is a short, packaged playbook the agent loads before it starts working: the traps to check, the order to work in, the things a finished answer must contain. All seventeen are open source in the repo. The full library:
Review and quality
- multi-model-adversarial-review · runs a second, independently vendored model against your own review pass and reconciles the findings.
- tiered-consultancy-review · a five-tier review ladder that takes a deliverable from rough draft to genuinely finished.
- pre-merge-validation-gate · defines what “done” actually means for a change and how to report test results without overstating them.
Systems and architecture
- architecture-contracts-as-law · keeps one merge-blocking source of truth for system invariants in sync with the code.
- multi-tenant-auth-reference · a ground-truth pattern for token kinds, role checks, and tenant isolation.
- llm-eval-harness-and-scoring-pipeline · locked scoring math, partial-failure handling, and prompt versioning for pipelines that grade LLM output.
Cost and safety
- ai-cost-tracking-and-guardrails · enforced call tracking, safe cross-provider fallover, and fail-closed rate limiting for LLM calls touching money or sensitive data.
- budget-aware-model-allocation · how to work deliberately when a token or rate budget is running low.
- config-and-secrets-hygiene · picking the right config layer and adding feature flags without precedence traps.
Deploy and infrastructure
- staging-to-prod-cutover-campaign · first-apply traps, the do-not-inherit config scrub, and separating go-live gates from infra bring-up.
- environment-and-build-hazards · the two-role database model, seed-data idempotency, and cloud-auth preflight for local development.
Debugging
- systematic-debugging-playbook · establishes ground truth before hypothesizing, plus the regression rule for multi-round fix loops.
- failure-archaeology · a “settled battles” record so nobody re-attempts an approach already tried and deliberately abandoned.
Process and change management
- git-change-control-for-agents · state verification before any git work, the dead-base PR trap, and working-tree discipline.
- multi-agent-batch-campaigns · wave planning and file-contention mapping for running a large backlog across parallel agents.
Documentation and compliance
- docs-of-record-and-arbitration · an explicit arbitration order for when project documents disagree.
- consent-and-regulated-data-reference · fail-closed defaults for consent, retention, and audit trails around regulated personal data.
Every skill here has its own card, with real excerpts from the skill file and a search box, on the skills library page.
The grid
Four Claude models. Three of them expose a reasoning-effort dial with five steps; Haiku runs at one fixed setting and serves as the small-model baseline.
| Model | Effort settings tested |
|---|---|
| Haiku 4.5 | no effort dialruns one way |
| Sonnet 5 | low · medium · high · xhigh (extra high) · max5 settings |
| Opus 4.8 | low · medium · high · xhigh · max5 settings |
| Fable 5 | low · medium · high · xhigh4 settings completed |
| Fable 5 at max | did not finishslot open · see below |
Methodology: every setting above ran the full 17-task suite twice, with the skills installed and without.
The pipeline, start to finish
From the frozen task suite to the final scored grid. Every setup walks the same path.
Methodology: most cells ran three times per side, with Fable's medium and extra-high settings and the Haiku baseline the single runs. Across the study the two judges disagreed on 4.6% of marks, 2.5% of marks went to the deciding judge, and none were left unresolved. The suite hash and every per-setup rate are in MATRIX.md in the repo.
How scoring works
Each task carries a short checklist of things a correct answer must do; a score is the share of checklist items hit. The study calls these checklist items must-hits. There are no style points: an answer either did each thing or it did not.
Every answer was scored blind by two Claude judges, one Sonnet-class and one Opus-class. They saw the two answers to a task in a random order and were never told which one had the skills installed. When the two judges disagreed on a checklist item, a fixed third judge, Claude Fable 5, settled it. Across the study the two judges disagreed on 4.6% of marks, 2.5% of marks went to the third judge, and none were left unresolved.
What each cell cost
Every run recorded its own cost as it ran. All in, the experiment cost $1,296.94: $1,034.05 running the arms and $262.89 judging them, with the cost of building the harness excluded. The table shows the average cost of one run, in dollars, for each model and effort setting: without the skills, with them, and the difference.
| Model | Effort | Without skills | With skills | Skill cost per run |
|---|---|---|---|---|
| Haiku 4.5 | no effort dial17 runs per side | $0.21 | $0.22 | +$0.01 |
| Sonnet 5 | low51 runs per side | $0.44 | $0.49 | +$0.05 |
| medium51 runs per side | $0.39 | $0.43 | +$0.04 | |
| high51 runs per side | $0.49 | $0.62 | +$0.13 | |
| xhigh51 runs per side | $0.81 | $0.80 | -$0.01cheaper with skills | |
| max51 runs per side | $1.20 | $1.18 | -$0.02cheaper with skills | |
| Opus 4.8 | low51 runs per side | $0.59 | $0.67 | +$0.08 |
| medium51 runs per side | $0.66 | $0.70 | +$0.04 | |
| high51 runs per side | $0.53 | $0.59 | +$0.06 | |
| xhigh51 runs per side | $0.72 | $0.75 | +$0.03 | |
| max51 runs per side | $0.89 | $1.00 | +$0.12 | |
| Fable 5 | low51 runs per side | $0.89 | $1.07 | +$0.18 |
| medium17 runs per side | $1.05 | $1.22 | +$0.17 | |
| high51 runs per side | $1.21 | $1.50 | +$0.30 | |
| xhigh17 runs per side | $1.41 | $1.62 | +$0.21 | |
| max | did not finishslot open | |||
Methodology: real recorded spend averaged per run, not list-price estimates, computed from cost-per-run-dataset.json, which ships next to this page. Loading the skills added between $0.01 and $0.30 to a run, and on Sonnet's two highest settings the runs with skills installed came out slightly cheaper than the runs without. The machine mix varied between runs, so treat these as observed spend, not a price benchmark. The full per-call ledger is in the repo.
The cell that got away
Fable at max effort is the one empty cell in the grid. It got too expensive to finish: the subscription pool it ran on expired before it did. The slot stays open by design: it is the pre-registered exploratory cell, permanently excluded from the confirmatory 15-cell matrix regardless of whether more of its runs ever complete. Nothing else on this page waits for it; the rules already cover the gap, as laid out in the full methodology.
The whole grid at once
Bar height is the score with the skills installed, on the shared 17 tasks. Drag to rotate; hover a bar for its numbers and run count. The toggles first add the runs without skills, then highlight what the skills added.
Hovering a bar highlights its effort level across models; scores are only comparable at the same effort level. The dashed square on the floor is Fable 5 at max, the run that got too expensive to finish.
| Effort | Fable 5 | Opus 4.8 | Sonnet 5 |
|---|---|---|---|
| low | +4.9avg of 3 runs | +8.3avg of 3 runs | +11.3avg of 3 runs |
| medium | +7.41 run | +11.8avg of 3 runs | +8.4avg of 3 runs |
| high | +2.4avg of 3 runs | +9.8avg of 3 runs | +11.3avg of 3 runs |
| xhigh | +2.91 run | +7.8avg of 3 runs | +11.3avg of 3 runs |
| max | not run · it got too expensive to finish | +8.8avg of 3 runs | +9.8avg of 3 runs |
Haiku 4.5, the one model with no effort dial, gets a single cell: the skills added +26.5 points (1 run per side). Cells here average three runs per side, except Fable at medium and extra-high, which are single runs, so on those two read the direction, not the decimals.
Methodology: all 15 completed setups plus the open Fable-max slot, scored on the 17 tasks every setup completed. The scores with and without the skills behind every bar are in the big table in The result. Source: MATRIX.md in the repo.
The full methodology
If you came to check my work, this section was written for you. The design in three sentences: three Claude models at five reasoning-effort levels each, plus an effort-invariant Haiku baseline, every configuration run cold and loaded (without the skill library installed, and with it) on the same frozen 17-task suite. Each model's endpoints, its lowest and top effort settings, ran three times per side; on Sonnet and Opus the interior levels did too, while Fable's two interior settings and the Haiku baseline stayed single runs. Two confirmatory hypotheses, frozen before any run, and one of them lost.
The lattice, fully specified
3 models × 5 effort levels + Haiku = 16 model-effort conditions × {cold, loaded} = 32 arms. The suite is 17 frozen primary tasks, hash-pinned at freeze: b378c79644280bb93fb8ac71d0cadcfe301fd15b226dfaec852b619a2aa1c890. Endpoint cells ran at 3 consumer repeats (three full runs of the model under test, per side), every repeat persisted independently with no shared session state; Sonnet and Opus ran at 3 repeats across their interior levels too, leaving Fable's medium and extra-high cells and the Haiku baseline as the only single-run cells. Plots and tables distinguish replicated means from single points, and the single-run cells get no uncertainty bars, on purpose: one run is one run, and an error bar around a single sample would read as precision the data does not have. Effort defaults were verified in smoke runs, never assumed; the values recorded in run-meta are authoritative.
Pre-registration mechanics
The hypotheses first, since the shorthand recurs from here on. H1: raising effort alone lifts a model's cold score by at least 3 percentage points (pp) between its endpoints. H2: the skill delta, the loaded score minus the cold score, shrinks by at least 3 points from the low endpoint to the top one. H1 and H2 are the only confirmatory hypotheses; H3 (whether skill value varies across task visibility tags) and H4 (the cross-model shrinkage comparison) are explicitly exploratory.
Frozen before any official run: those hypotheses, the 3pp minimum effect, endpoint-only decision rules (interiors can never touch H1 or H2 under any circumstance), paired exclusion (either arm invalid kills the whole cold-loaded comparison), a common complete-case set per model column over endpoint cells only (each model's two endpoints are scored on exactly the tasks valid in both, so nothing else can shift the comparison), blocked randomization stratified within model column (run order shuffled in blocks, separately for each model), identical max_tokens across all 32 arms, truncation invalidates the run (one retry, then excluded), frontmatter preflight (the metadata headers of the task and skill files validated before any run), cold-arm isolation checks (the cold runs must not be able to see any skill file), and cross-model-fallback invalidation (a run that silently fell back to a different model is thrown out).
I declared my prior at freeze: H1 and H2 supported, H3 uncertain. H2 then failed everywhere. I could not have fit the rule to the result: the rule predates the result and says so in a public hash.
The judging stack, mechanism by mechanism
Every mark is scored by a blinded, order-randomized two-judge panel: one Sonnet-class and one Opus-class model, IDs pinned, both at medium effort, never varying across cells. When the two judges disagree on a mark, it goes to a pinned third adjudicator, claude-fable-5 at medium, which sees only the disputed criterion and the two blinded report slots, a narrower input than a full judge gets. It scores the binary mark independently, the two-of-three majority is final, and disputed marks never leave denominators. Everything persists: both original marks, evidence quotes, the adjudicator's mark, the majority result.
The rates, published: 122 of 2652 marks disagreed (4.6%); 135 of 5304 report-slot marks were adjudicated (2.5%); 0 unresolved. The per-cell breakdown is in the repo's MATRIX.md; Haiku's cell was the noisiest at 13.2% mark disagreement.
The residual limitation is that Claude models judge Claude outputs. The pre-registered mitigation: the committed judge inputs are public, plus an exploratory Codex cross-vendor concordance sample of 50 comparisons, which has now been run: Codex (gpt-5.6-terra at high reasoning effort) agreed with the panel-final marks on 97.2% of marks (383 of 394), 0 unscorable. It is exploratory only and moves no verdict.
Disclosures
One blinding exception exists in the committed data, found by a post-run audit and disclosed rather than papered over. The scrubber banned every form of a task's own skill name but not the names of sibling skills, and skills sometimes cross-reference each other in their own text. Exactly one of the 456 committed judge inputs was affected: the Haiku cell, task aicg-t2, report 1, where the loaded answer quoted its skill's cross-reference to a sibling skill and the sibling's name survived the scrub. A judge reading that one input could infer which arm was loaded. The scrubber and its leak verifier now ban all seventeen skill names in every separator and concatenation variant, with a test covering the exact committed case. The impact is bounded: one judge input of 456, in the contextual Haiku cell, outside every confirmatory hypothesis. No H1 or H2 number moves.
Concordance selection wording. The pre-registration named the Codex sample's selection only loosely as “hash-parity”, without a full construction. The implemented rule is deterministic and is disclosed in the repo's concordance report as the implemented interpretation rather than claimed as literal pre-registered wording: digest = SHA256(suite hash + cell + filename), sort all committed judge-input comparisons across the 15 scored cells by digest, take the first 50. Rerunning the selection reproduces the identical 50.
Sanitization. Before the repository's first public release, nine committed results files had absolute local filesystem paths redacted by mechanical, byte-level string replacement; no mark, judgment, score, task content, or model output beyond those path strings was altered. One affected file is a blinded judge input that had already been scored before the redaction, so the recorded marks were produced against the pre-redaction text, which differs only in the redacted path bytes. The full record is results/SANITIZATION.md in the repo.
The floor-exclusion rule
The rule, exactly: if more than one third of a comparison's marks are knocked out through the judge-failure path (after the adjudicator amendment made disagreement itself non-destructive), the entire paired cell is excluded and reported, so no verdict can rest on a task judged only on its easiest surviving marks. It exists as the failure backstop, not the disagreement path. It tripped zero times in the final data. There is also a pre-registered column-level degradation threshold: if more than 10% of a column's endpoint comparisons are excluded, the column reports degraded and issues no verdict. Honesty note: that threshold was never implemented in harness code; it was checked by hand on the final data, and the check came back clean. No column degraded: all three models came in at 0 of 102 endpoint comparisons excluded, well under the 10% rule. A second pre-registered hand check, flagging any endpoint task with fewer than 2 surviving repeats, was also performed: zero tasks tripped it on any endpoint cell.
The two harness bugs
Both were found the night before the final sweep, by stacked adversarial review passes: a Sonnet recon pass, then an Opus-high verification pass, then an Opus-xhigh (extra high) completeness critic. Both were fixed and tested in commit 27d44f8, with 255/255 tests green plus a live matrix-report smoke over the 10 then-complete cells before any real report was generated.
Bug 1, the Fable endpoint override. The publicly posted amendment moved Fable's confirmatory endpoint from low-vs-max to low-vs-high, because fable-max could not finish on the expiring pool (2 of 102 consumers). But report.py's _endpoint_cells() hardcoded low-vs-max for every model. Here is the failure mode. The harness would never have crashed. It would have silently reported Fable's H1 and H2 as "not evaluable", or scored the wrong pair, forever, while every test stayed green. The fix is a per-model confirmatory-endpoint override plus disclosure notes wherever the amended pair renders, including the H4 table, where the first version of the fix itself introduced an undisclosed relabeling that the third review pass caught. A fix that quietly mislabels one table is exactly the kind of thing that ships when only one model looks at it.
Bug 2, complete-case scoping. The complete-case set feeding H1, H2, and retention was computed as one global intersection across every cell passed to --matrix-report, contradicting the frozen prereg text ("within that model column... endpoint cells only, interior cells never enter this set"). The consequence is worth spelling out: one model's task exclusion could silently shrink another model's confirmatory delta, and including the empty fable-max directory would have collapsed every model's H1 and H2 to "not evaluable". The fix is a per-model-column endpoint-only intersection (_endpoint_complete_case), with the global set retained where it is legitimately the right denominator, the cross-model descriptive views.
The process point: neither bug was found by tests. Both were found by paying a second and a third model to re-read the code against the frozen prose. Budget for that.
The gate that already existed, and where it came from
Before every column, the harness required a live-smoke-green gate: a one-task end-to-end smoke run, judges included, green before the column could start. That gate is itself a process correction. It was born earlier in the campaign, when a judge-path bug slipped past testing and was caught late. Here it is, prominently, rather than buried. The bug: the judge invocation was broken on the live CLI path in two ways at once. judge.py handed the CLI a file path where it needed the JSON schema inline, and the schema file carried a $schema key the CLI rejected. Every unit test stayed green through both, because green unit tests say nothing about the CLI contract; a live run would have burned the full consumer spend and then died at judging. The verify stage of the harness review found both bugs, the fix stage was never given the verify stage's findings, and the breakage survived two consecutive "all fixed" reports before it was fixed, before the freeze, with the live smoke green after. Since then, every findings source feeds the fix stage, and no "all fixed" claim stands without a live one-task smoke, judges included.
Execution engineering
Two machines, expiring weekly pools. Consumers, the model runs under test, raced first with --defer-adjudication (finish the runs now, settle judge disagreements later); judging lagged on committed inputs. Handoff discipline: raw artifacts only (consumer outputs, judge inputs and outputs, order keys, run-state, run-meta), never the second machine's own scores.json, always recomputed centrally, suite hash verified on every merge. Idempotent resume semantics (skip-if-persisted at the consumer, judge, and adjudication layers) meant account-wide usage-limit pauses cost wasted call attempts and nothing else.
One disclosure, the resume-identity trap: on the opus-medium merged directory, a same-config rerun against a directory whose run-state only knew 8 of 17 tasks would have silently rebuilt the summary down to 8 tasks. The workaround ran adjudication in scratch directories and copied only the new adjudication files in.
Costs: all in, the experiment cost $1,296.94, of which $1,034.05 went to running the arms and $262.89 to judging them ($207.72 for the two-judge panel, $55.17 for adjudication). Every figure is computed directly from the recorded cost of every API call on disk, not estimated. One cell's records exist in two overlapping ledgers (a handoff merged onto the primary disk for central scoring), so the total deduplicates per run before summing: $1,296.94 of gross ledger entries minus $0.00 of double-counted records lands on $1,296.94 exactly. Building the harness is not in that number. The per-cell penny math is in results/matrix/NUMBERS.md in the repo.
Full results
| Model | cold(low) | cold(max) | Difference (pp) | Verdict |
|---|---|---|---|---|
| claude-fable-5 | 94.1% | 97.6% | +3.5 | directionally supported under the pre-registered rule |
| claude-opus-4-8 | 87.3% | 91.2% | +3.9 | directionally supported under the pre-registered rule |
| claude-sonnet-5 | 84.8% | 89.7% | +4.9 | directionally supported under the pre-registered rule |
| Model | D(low) pp | D(max) pp | Shrinkage (pp) | Verdict |
|---|---|---|---|---|
| claude-fable-5 | +4.9 | +2.4 | +2.5 | not supported under the pre-registered rule |
| claude-opus-4-8 | +8.3 | +8.8 | -0.5 | not supported under the pre-registered rule |
| claude-sonnet-5 | +11.3 | +9.8 | +1.5 | not supported under the pre-registered rule |
| Model | R |
|---|---|
| claude-fable-5 | 0.490 |
| claude-sonnet-5 | 0.867 |
| claude-opus-4-8 | 1.060 |
Fable rows are low vs high per the posted endpoint amendment, everywhere they appear. All figures complete-case (the shared task set), must-hit weighted (every checklist item counts once, equally), small-n directional. Full cell tables, the per-skill matrix, invalidation rates, and the task-cluster bootstrap (a descriptive spread check that resamples whole tasks) live in MATRIX.md and matrix.json in the repo.
Basis note: every table and chart on this page is scored on one task set, the 17 tasks valid in every cell, so the confirmatory tables and the grid-wide tables read the same numbers for a given model. An earlier version of this page split them across two slightly different task sets; that gap closed when a censored run was recovered, and the two now coincide.
H3 (visibility-tag heterogeneity) is exploratory prose only; no computed contrast exists in the harness. H4 is a directional-only cross-tier shrinkage comparison, carrying the same Fable low-vs-high disclosure. Haiku's +26.5pp is contextual, outside all effort-trend hypotheses. The interior levels are complete: on Sonnet and Opus every effort level now carries three repeats per side, so the medium, high, and extra-high cells are replicated means rather than the single runs they began as. By construction none of them can move H1 or H2 anyway, since both verdicts read only each model's two endpoints.
Limitations, once each
Small n everywhere: 17 tasks, most cells three repeats per side, single runs only on Fable's medium and extra-high settings and the Haiku baseline; every rate is directional. Judging is same-vendor, mitigated but not eliminated by the two-judge panel, the pinned adjudicator, the public judge inputs, and the Codex cross-vendor concordance sample (exploratory; now run, n=50, 97.2% concordance with the panel-final marks, 0 unscorable). The suite was authored by the skill author, so the construct-validity claim is limited to "skills improve recognition and handling of the failure classes they were designed to address", never general software-engineering improvement. The loaded-arm estimand is the total product effect (install plus instruction plus content), not pure content. Parked for future runs: out-of-distribution tasks authored skill-blind, and a placebo-document arm.
What did not run
fable-max is the pre-registered open/exploratory cell: it holds no scored data, by design it never joins the confirmatory 15-cell matrix regardless of outcome, and Fable's confirmatory endpoint moved to high per the posted amendment. Beyond that, the parked items named in the limitations above, out-of-distribution tasks authored skill-blind and a placebo-document arm, did not run either.
A per-cell explorer for this study is built from the same machine-readable matrix file the report is generated from, with every cell browsable by model and by reasoning-effort level, including the max column.
The harness that emits the matrix is in the repo, so you can grade your own.
Reproduce it
The repo carries the frozen pre-registration, the committed judge inputs, and matrix.json. Run the suite on your own keys. Re-judge my judge inputs on a model I do not control. Tell me where it breaks: github.com/HamzaYM/reliable-ai-skills.
One more thing worth saying plainly. This entire experiment, the harness that ran it, the judging panel that scored it, and the orchestration across two machines that kept it moving, was built and run on Claude Code, by Claude models, with me directing. The judge inputs are public on purpose: anyone (and that includes anyone at Anthropic) can re-score them with a different model and check whether the results hold up. The instrument can also run against models I have not tested, to check whether the skills hold their effectiveness as effort scales past what I can measure. If you have ideas, reach out. If you want to extend the library or the task suite, pull requests are open. And if you run this against your own stack, send me your results and I will add them here.
The paper trail
- The repo: the harness, the 17 skills, the frozen task suite (
eval/tasks/golden-suite.jsonl), and the committed judge inputs. - The full matrix report:
results/matrix/MATRIX.mdandmatrix.jsonin the repo. - The pre-registration (
effort-sweep-preregistration.md) and the posted 2026-07-10 endpoint amendment (effort-sweep-amendment-2026-07-10-fable-endpoint.md). - The Codex concordance report:
results/concordance/CONCORDANCE.md. - The per-cell explorer: explorer/, built from the same matrix.json the report is generated from.
- The publicly posted pre-registration hash: gist.github.com/HamzaYM/3554b04c78b918d152cdaa8b3c705987.
I predicted the skill advantage would shrink at higher effort. It held the entire way up.