Field notes

Before a case draft for a graduate course session went in front of real people, I ran it past a fake room first.

The setup: seventeen persona agents, each one seat. A risk officer, a CFO, a platform engineer, a policy staffer, and so on, each defined by a professional background, a temperament, and nothing else. What a seat knows is a design decision, not an accident of context. The seats are spread across three model tiers on purpose. Seventeen copies of the same model give you seventeen flavors of the same read; mixing tiers gives you the skimming executive and the person who checks the exhibits twice, which is what a real room contains.

The protocol is the part I'd defend hardest. Every seat reads the draft blind and returns a structured vote. No cross-talk, no anchoring on the loudest agent. Then the votes are revealed and the room votes again. The gap between round one and round two tells you whether the case moves a room or just survives one.

I ran the loop the way you'd playtest a board game: full run, read the transcript, revise the draft, run it again. The panel caught things I would have found out the hard way in class, like which exhibit the fast readers skipped and where the discussion stalled with nowhere to go.

The carry-away rule: any room where people read something and then decide is a topology. If you can cast the seats and wire the protocol, you can run the room before it exists, and revise while revision is still cheap.

The intake conversation on a healthcare intake platform I work on runs on a mid-tier model. The cheap, fast tier costs a fraction as much, and the obvious question kept coming up: why not switch?

I didn't switch. I built a shadow lane instead. Every intake on staging kept its normal model, and a second extractor ran silently alongside it on the cheap tier. Both results landed in a durable log with agreement scoring, so I could see exactly where the small model matched the big one and where it drifted, on real conversations, at zero risk to anyone filling out a form.

Once the shadow data looked promising, I ran a proper trial: a generation by extraction matrix, four cells, about 48 fresh staging intakes. The mixed cells held up. The all-cheap cell did not; half its intakes got stuck or were abandoned. That one afternoon of evidence settled a debate that had been running on vibes for weeks. The default stayed where it was, documented as a decision rather than an assumption.

One piece did ship: a fast path on the cheap tier with an automatic cascade. If the small model repeats an identical question, the turn escalates to the bigger model. The small model gets the easy work and a tripwire, not the keys.

The rule I took away: never promote a cheaper model on benchmarks or price. Run it in shadow against production traffic, score the agreement, and let it earn each role separately.

At a business-aviation operator, every trip request has to clear the same gauntlet before anyone commits an aircraft: can the passengers actually enter that country, is the airport open and staffed when they land, is the tail out for maintenance, does the crew have legal duty time left, will they arrive in daylight where that matters. Dispatchers carry this checklist in their heads. We built it into software.

The engine runs a set of process checks against each trip: customs and immigration (including the edge cases, like domestic legs where customs and visas don't apply), aerodrome status and clearance hours, maintenance downtime that overlaps the trip dates, scheduling conflicts, and crew duty time computed across all legs, adjusted for the regulatory category the flight operates under. The checks run in parallel, and each one comes back with a status and its reasoning, not just a red or green light.

Two design calls mattered more than any individual check. First, thresholds came from the operations floor, not from us; two legs of the same flight only count as a conflict when the gap is under 45 minutes, because above that the schedule absorbs it. Second, overrides are first-class. A dispatcher can wave a check through, and the engine reruns everything afterward so the override is a recorded decision on fresh data, not a silenced alarm.

The lesson: a feasibility engine earns trust by encoding the questions experts already ask, showing its work, and leaving the final call with the human.