baseline-experiment-80

gpt-5.6-luna · 13620 runs · 10 arms · 460 tasks · controlled

The arms

Derived from each run's recorded axis assignment, not from the preset name — so an arm is described as what it actually was, including any sweep override.

The verdict

Every dimension is rescaled 0–1 across the packaging arms, oriented so higher is always better, then weighted. Controls are shown but never scored — Z0 has no tools and Z1 is handed the answers, so neither is something you could ship.

Weights: success 0.35 · abstention 0.15 · harm 0.25 · cost 0.15 · time 0.10. Change them with --weights harm=1 to see, say, the safety ranking alone. † marks a control, excluded from scoring.

Lift over baseline

Whiskers are Wilson 95% intervals. Hatched bars fall inside the minimum detectable effect and are not findings.

Where runs end up

Truncated runs hit max_turns. They are shown here as their own segment and excluded from success rates — a budget failure is not a wrong answer.

Success by task class

Whether the arm ranking reverses between reads and writes (RQ4).

Answer or abstain

A success rate cannot separate an agent that fabricates an answer from one that gives up. Both fail; only the first is dangerous.

F1 ignores true negatives, but a correct refusal is a first-class success here — read balanced accuracy and MCC alongside it. MCC is the honest single number on skewed classes: an agent that answers everything scores high accuracy and MCC near zero.

Cost

Down and to the right is better: cheaper per success, more successes.

Token components

Reported as components and never totalled: stateless MCP moves cost from session setup to a per-call tax that scales with call count, and a total hides that.

How each arm behaved

These explain why the success rates came out the way they did. Ranked by how much each one separated the arms — a metric that came out the same everywhere explains nothing about why one packaging beat another.

No variation across any run on: forbidden_call_rate = 0, hallucinated_endpoints = 0. Charting these would imply a difference that is not there.

Every metric above is unvalidated: its relationship to ground truth has not been established on the controlled rig.

Contrasts

Paired within core, so per-core difficulty cancels. Only pre-registered contrasts can reach significance; exploratory ones are shown and labelled but never corrected.

Operation ledger

Which parts of your API agents lean on and misuse — so you know what to document, hide, or redesign. Rates, not raw counts. Not for declaring a packaging winner (that is the scorecard above).

Controls with no target calls (omitted): Z0, Z1.

Gold path (navigation + terminal writes): get_episode, get_series, list_episodes, list_series, list_studios, patch_episode, append_episode_tag, archive_episode. Off-path = called on an answerable task but not on that path.

A. Over-touch

Called more than the gold path expects — candidates to document or hide. Bar length is excess share of all target calls.

B. Stumble by kind

Separate failure modes — not one blended misuse score. Ranked by rate × volume so a rare 100% miss does not outrank a common problem.

Distractors

High off-path and almost never on gold — strong hide / docs candidates.

C. Resource families

D. Per-arm cards

What each packaging leaned on and where it stumbled — headlines for skill/docs edits, not winners.

E. Skill / discovery contrasts (off-path)

Fixed pairs only (skill on/off, eager vs meta-tools). Negative Δ means the right-hand arm improved. Descriptive — not confirmatory, no MDE.

A1 → B1-auth authored skill on eager MCP

No swing ≥ 5 pp on shared ops.

A2 → B2-auth authored skill on meta-tools

No swing ≥ 5 pp on shared ops.

D1 → D2-auth authored skill on code sandbox

No swing ≥ 5 pp on shared ops.

A1 → A2 eager-all vs meta-tools discovery

No swing ≥ 5 pp on shared ops.

• 20476 discovery meta-tool calls (search/describe/invoke) omitted from the charts above
• 20256 shell/code calls resolved by parsing transcripts — approximate, not a server request log
• Volume is not blame; HTTP 200 can still harm; unanswerable thrash is abstention, not an outage

Details

Answer or abstain

Precision falls when an arm answers what it should have declined; recall falls when it declines what it should have answered. Accuracy alone cannot tell those apart, which is why MCC sits beside it.

Effort per run (mean)

562,087,319 tokens across 13620 runs. Token counts are not comparable across providers — compare dollars per success, not tokens.

Outcomes

* difference falls inside the MDE.

Success by class

Harness errors

Glossary

arm

One way of packaging the API. The thing being compared. Every arm gets the same tasks against the same API. The only difference is how the API is presented to the model — as MCP tools, as written docs it curls, as a code library, and so on.

task

One question or instruction given to the model.

core

One navigation problem, reused across five task types. The same 'find episode X' problem appears as a read, three kinds of write, and a bulk operation. Because the difficulty of finding the target is identical across all five, any difference between them is caused by what happens AFTER it is found — not by one being harder to locate.

run

One arm attempting one task once.

Z0

Control: the model gets NO tools at all. It measures what the model already knew before you gave it anything. On a made-up API this should be ~0 — if it is not, the API leaked into the model's training and every result is void. On a real API it is often high, and then the tools have to beat it to be worth having.

Z1

Ceiling: the correct API responses are handed over up front. The model does no tool use — it just reads the data and answers. If Z1 scores 80%, then 80% is the best any packaging could do, and an arm at 75% is nearly perfect rather than mediocre. Without it you cannot tell a bad wrapper from an impossible task.

success rate

Share of tasks the arm got right. Excludes runs that ran out of turns — those are a budget problem, not a wrong answer.

lift

Success rate MINUS Z0's. How much the packaging actually added. An arm at 67% sounds fine until you learn Z0 scored 83% — then it is 17 points WORSE than giving the model nothing. Raw success can flatter; lift cannot.

pp

Percentage points — the gap between two percentages. 70% minus 55% is 15 pp. Writing '15%' would be ambiguous, because 15% OF 55% is 8.25%. 'pp' says plainly that one was subtracted from the other.

truncated

The run hit its turn limit before answering. Not counted as wrong — it might have been one step away. Reported separately, because a high truncation rate is its own finding: the arm is spending its budget without finishing.

infra-error

The harness, provider or machine broke — the arm was never measured. A full disk, an expired API key, a provider timeout. Excluded from every rate, because counting a full disk as a packaging failure makes an arm look worse than it is. Carries an error_kind saying which, and --resume re-runs these cells rather than skipping them.

TP / true positive

Answerable task, answered correctly.

FP / false positive

A task with NO valid answer, and the model made one up. The dangerous failure. About 15% of tasks are deliberately unanswerable to catch exactly this.

TN / true negative

Unanswerable task, correctly declined, catalog left intact.

declined-but-clobbered

Unanswerable task: refused in the transcript but mutated server state. Not a true negative. Kept separate from false-positive (fabricated an answer) because the writeup needs both failure modes.

FN / false negative

Answerable task, and the model failed or gave up.

precision

Of the answers it gave, how many were right. Low precision means it answers when it should stay quiet. TP / (TP + FP).

recall

Of the answerable tasks, how many it actually answered. Low recall means it gives up too easily. TP / (TP + FN).

specificity

Of the unanswerable tasks, how many it correctly refused. Low specificity means it invents answers or refuses while writing. Also shown as 'abstain' in the ranked tables. Declined-but-clobbered counts against this number — a mutating refusal is not a correct decline.

abstain / abstention

The same number as specificity, under a plainer name. Scored as its own dimension in the composite rather than left inside the success rate: only ~15% of tasks are unanswerable, so an arm that fabricates on every one of them loses at most 15 points of success — far too little for the failure that matters most.

F1

Precision and recall combined into one score. Useful as a summary, but it ignores true negatives — so an agent that correctly refuses gets no credit for it here. Read specificity too.

accuracy

Share of ALL decisions that were right. Misleading when most tasks are answerable: a model that answers everything scores ~85% accuracy while being maximally unsafe.

balanced accuracy

The average of recall and specificity. Treats 'answered correctly' and 'refused correctly' as equally important, which fixes accuracy's blind spot.

MCC

One number from -1 to +1 summarising the whole 2x2. +1 is perfect, 0 is no better than guessing, negative is worse than guessing. Unlike accuracy it cannot be gamed by always answering — that agent scores 0. The honest single number when unanswerable tasks are rare.

silent failure

Wrong, and stated confidently rather than hedged. Being wrong is bad; being wrong and convincing is worse, because nobody downstream will check.

turns

Round trips between the model and the tools.

calls

Individual API operations the model invoked.

static tokens

Tokens spent before the model does anything. Tool schemas plus any skill or docs. An arm that loads 200 operation schemas pays for all of them on every single task.

cached

Share of input tokens billed at the cheaper cached rate. Repeated prefixes get discounted. Writing the cache costs MORE than fresh input, so it only pays off from the second run onward.

$/success

Total spend divided by tasks solved. The comparable figure — dollars, not tokens, because different providers count tokens differently.

harm

Runs that destroyed data or attempted a forbidden operation. Counts blocked attempts too: on a real API there would be no guard, so the attempt is the finding.

SCORE / composite

Every dimension rescaled 0-1, then weighted into one number. A decision aid, not evidence. It will always produce a ranking, including one made entirely of noise — which is why the winner is printed with whether its lead is larger than the MDE. Read the score to shortlist; read the interval before acting.

weights

How much each dimension counts toward the score. Default: success .35, abstention .15, harm .25, cost .15, time .10 — so safety (.40 combined) outweighs thrift (.25) and an arm cannot buy its way to the top by being cheap and dangerous. Override with --weights; they are always printed beside the table.

candidate arm

An arm eligible to win: a packaging you could actually ship. Z0 and Z1 are excluded. They anchor the scale rather than competing on it, and letting the ceiling arm into the range would squash every real arm into the bottom of the scale.

spread

How much a behavioural metric separated the arms. Decides which metrics get charted. Deliberately not a coefficient of variation: one arm at 0.15 against eight at zero would score huge and outrank a metric running 0 to 2.8 across four arms — that separated one arm, not the arms.

discovery overhead

Calls made before the first one that returned usable data. A discovery arm pays this by design; the question is whether it earns it back in fewer wasted calls later.

wasted calls

Responses whose content never reaches the final answer. Data fetched and then ignored. Every one filled the context window for nothing, and it is usually the clearest sign that a surface returns more than the task needs.

redundant calls

The identical request issued more than once in one run. The model is not retaining what it already fetched and is paying twice. High counts usually mean the response was too big to hold onto, not that the model was careless.

payload efficiency

Of everything the API returned, the share used in the answer. Low means responses are far larger than the task needs. Fixing it is an API change — sparser responses — not a packaging change.

argument validity

Share of calls the API accepted rather than rejecting as malformed. Low means the schema is not telling the model how to call it. This is the number a richer schema should move.

error recovery rate

Of the calls the API rejected, how many it then got right. The mechanism by which better error messages would beat shipping a skill: an error only helps if it says enough to act on.

hallucinated endpoints

Calls to operations that do not exist. The model invented them rather than reading the surface — evidence that discovery is failing, not that the model is careless.

peak context tokens

The largest the context window got during the run. What would have to fit if you pointed this packaging at a bigger problem. An arm near the limit here will break on a harder task.

turns to first productive call

How long before the model got anything useful out of the API. Measures how quickly a packaging becomes usable, which is not the same as how well it ends up performing.

95% CI

The range the true value plausibly sits in. A wide interval means too little data to say much. If it spans zero, the difference could just as easily be nothing.

MDE

Minimum detectable effect — the smallest gap this run could spot. With few tasks, only huge differences show up. A gap smaller than the MDE is not a finding, however suggestive the bar looks.

p value

Roughly: how easily chance alone could produce this gap. Below 0.05 is the usual bar. It says nothing about whether the difference is large enough to care about — that is the CI's job.

Holm correction

Adjusts p values for testing several things at once. Test six comparisons and one will look impressive by luck. This raises the bar so that does not happen.

confirmatory

A comparison declared BEFORE the run, in code. Only these can be called significant. Deciding what to test after seeing results is how noise becomes a headline.

exploratory

A comparison looked at afterwards. Reported and labelled, never called significant. It can suggest the next experiment; it cannot support a claim.

paired within core

Arms compared only on problems they both attempted. A hard problem drags every arm down, so comparing them on the same problems cancels that out.

refusing to pool

Results that must not be averaged together. Different model, different protocol version, or a hand-written vs generated skill — averaging those answers a question nobody asked.

unvalidated

The metric is computed, but not yet checked against ground truth. It describes what happened. It is not yet evidence of quality.

reference run

The first directory you passed. Every delta is measured against it. Swapping the order flips every sign. The reference is named in the header so a reader never has to guess which way a difference points.

Δ / delta

This run minus the reference, in the unit the difference is in. Rates become percentage points (pp), never a percentage of a percentage. An absent number is n/a, never zero — a blank that reads as zero is a claim nobody measured.

shared arm

An arm every compared run actually ran. Only shared arms enter the head-to-head. An arm two runs out of three have would leave a blank in a delta column, and a blank reads as a zero.

not compared

Arms missing from at least one run, shown standalone. They are excluded from every delta rather than shown with a blank cell. Their numbers are still reported — just not subtracted.

not recorded

This run's ledger cannot say what the parameter was. Not the same as equal. Two runs that both cannot say might still have differed, and reporting that as a match is the lie the sentinel exists to prevent. Common on ledgers written before a manifest field existed.

same world

Both runs were given the same generated tasks. Core ids are positional; core-000 in one world is a different problem from core-000 in another. Pairing across worlds would produce a tighter interval than the data supports — the most dangerous direction for an error to go.

unpaired comparison

A cross-run gap computed without pairing within core. The common case when two runs used different seeds, cores, or difficulty. Per-task difficulty no longer cancels, so the gap may be biased rather than merely noisy. Always labelled, never called significant.

cross-world delta

A difference across a pooling boundary — model, MCP revision, skill condition, or report class. The rates are real; their difference is not a finding. Marked ‡ and never called notable, however large the gap.

cross-run MDE

The smallest gap these two samples could resolve, in points. Larger than either run's own MDE: both measurements carry error and errors add (≈√(a²+b²)). An approximation, not a derivation from the power model, and labelled as one wherever it appears.

parameter diff

What differed in the setup before any result is shown. A delta table read without knowing the runs used different difficulty is worse than no table. Differs means two runs recorded different values; not recorded somewhere is uncertainty, not disagreement.

confidence interval

The range the true difference plausibly sits in. For unpaired cross-run gaps this is a Newcombe hybrid-score interval, not the textbook Wald form — Wald collapses at the edges and makes a 2-for-2 smoke arm look like a finding.