The design
A 14-agent discovery pipeline pulled 26 candidate skills from two production codebases; 8 of them went through an order-blinded A/B across 50 pre-registered must-hits in 16 tasks. Eight skills, two tasks each.
A must-hit is a pass/fail expectation written in advance. The output either meets it or it does not, and each task carried its own short list, 50 across the suite.
Two words in that design carry the weight. Pre-registered means the pass rule and every expectation were written down before any skill file existed, so the test could not drift toward the result I wanted. Order-blinded means judges saw the two reports for a task in an alternating order and were not told which arm was which.
Every task ran twice. In the cold arm, an agent worked the task with no skill loaded. In the loaded arm, an agent worked the same task with the skill loaded. A judge then graded each pair against the task's must-hits.
The harness this ran on is stage 03 of the assurance chain, the standard the client work passes. That page describes the instrument. This page is what happened when I pointed it at my own tools.
The numbers
Fifty must-hits, graded in both arms.
Task must-hits rose from 26/50 to 46/50, and all 8 tested skills cleared the pre-registered rule. The 26 sources were distilled into 17 portable skills, released as Reliable AI Skills, open source, with the harness included so you can grade your own.
The rule itself was fixed in the plan before the first run: a skill passes only if its loaded runs beat its cold runs and clear at least two thirds of their must-hits. Four tasks tied between arms; the gains came from the rest.
A bigger, pre-registered experiment followed: it tested whether more reasoning effort erases what the skills add, across three Claude models at five effort levels each, plus Haiku as a fixed baseline. It is complete, and the answer is on its own page.
Read the effort lattice →The timeline
The whole study ran on 2026-07-02, and the order of the timestamps is the point: the plan came first, and no skill file existed when it was written.
2026-07-02 · all times UTC
What this does not show
The study has limits. They are stated here, plainly, because a result you cannot bound is a result you cannot trust.
The judging was order-blinded, not content-blinded. The order key was withheld from the judges. The content was not scrubbed. Consumer reports included an actions section, and some judge verdicts noted that a report had consulted a skill. The next version of the harness judges outputs only, so that leak closes.
The released library is also not a straight copy of what was tested. The honest accounting: 26 repo-specific skills were consolidated into 15 generalized skills (19 merged down to 9, 6 carried 1:1, 1 dropped as product-specific, 2 largely absorbed into merges), plus 2 promoted from a separate personal library, for 17 total.
That has a consequence worth saying outright. 5 of the 17 released skills trace to A/B-measured sources. The measured deltas attach to the source skills as tested in their home repos, and the released rewrites carry the lineage, not a fresh measurement.
What comes next
That next run has since happened: the effort lattice, pre-registered the same way, judging outputs only, and measuring the released skills as shipped rather than their sources. Its numbers live on its own page.
The method and the harness are in the repo, so you can grade your own; the sequel ships with its own per-cell explorer.
The study on this page ran in a single configuration. The follow-up tests the released skill library across three models and five reasoning-effort levels, plus Haiku as a fixed baseline, in a pre-registered lattice, now complete, on its own page.
Read the lattice study →The library the 17 skills landed in, and the rest of the artifacts behind the lab, are on the proof page.
The question is never whether it looks right. It is what the evidence says.