The system
How the system worksThe DirectoryDispatchThe LedgerAI modelsAI agentsYour installationThe appThe research libraryWhat changes for your businessUse it from your softwareEverything it can do
Messaging
iMessageWhatsApp
Ads and leads
AdsCreativeLeads
Devices
Devices
Industries
RealtorsRestaurantsHome services
Explore
How it worksPermissions and limitsCompareSet upQuestionsText me

Our error rate looked three times better than it was, because of what we refused to count

An external cold audit read this site's adjudication numbers the way an adversary should, and found an arithmetic tension nobody inside the build had published: The known-answer probe suite has fourteen items. On three of them — P05, P07,

The finding, as it arrived

An external cold audit read this site's adjudication numbers the way an adversary should, and found an arithmetic tension nobody inside the build had published:

The known-answer probe suite has fourteen items. On three of them — P05, P07, P09 — the entire five-model panel was wrong: zero correct out of five, three separate times. Yet the published configuration table reports a five-channel floor of one undetected-wrong item in fourteen: 0.071, naming P07 as the sole survivor. If three items were unanimously wrong, why does only one survive every configuration?

The reconciliation was in the fine print. Two of the seventy findings were malformed — one confirmed at the receipt level as kimi-k2.6 returning UNPARSED on P05 — and were excluded from the configuration statistics, because a non-finding is not a rating. That exclusion is a defensible scoring decision. But it has a mechanical consequence the report did not state: a malformed finding forces the gate to escalate rather than emit. An unparseable output on an item the panel would otherwise have answered wrongly converts an escaped error into a human referral. On at least one, and possibly two, of the three unanimously-wrong items, the assembly was rescued not by diversity, not by the gate's design, but by a model failing to produce parseable output.

The headline number — five channels drive undetected-wrong down to 0.071 — rests in part on accidental parse failures. Take the rescue away and the floor bound is 3/14 = 0.214, roughly triple.

What is confirmed and what is inference, exactly

Confirmed, at the linked surfaces:

  • The exclusion policy exists and is stated on the probe report: 2 of 70 findings malformed, excluded from configuration statistics, retained in per-model rates.
  • P05, P07 and P09 were each 0/5 — printed per item, with the declared expected verdict and the reason it is correct.
  • kimi-k2.6 returned UNPARSED on P05 — the receipt caption says so.
  • A malformed finding cannot be emitted; the gate's only move is escalation.

Not yet resolved: which item the second malformed finding landed on. If it landed on P09, both rescues sit on unanimously-wrong items and the 0.214 bound binds tight. If it landed on an item the panel had right anyway, one of the three unanimous misses escaped by some other route and the accounting needs a different correction. The per-item receipts settle this and reading them is open work, stated here as open work.

Why the rescue is genuinely double-edged

It would be too quick to call this only an embarrassment. Escalating on malformed output is correct behaviour — a gate that emitted anyway, or guessed, would be indefensible. The assembly did, mechanically, the safe thing: faced with a channel that produced garbage on a question where every functioning channel was confidently wrong, it declined to answer. In the field, that outcome — a human looks at P05 — is strictly better than the alternative the other channels were unanimously offering.

The defect is not the behaviour. The defect is the bookkeeping: crediting that outcome to the assembly's measured error floor without disclosing that the mechanism was luck. A parse failure is not a safety property, because it is not reproducible on demand — the next run of P05 may parse cleanly and emit the wrong answer five-for-five. A floor propped by accident holds until the accident stops happening, which is precisely the kind of number that fails exactly when relied upon. The honest statement is now on the report: 0.071 is the floor under the stated exclusion policy; 0.214 is the bound under the accounting that treats rescues as escapes; a reader pricing a consequence should know which one they are holding.

The general lesson: an exclusion policy is a safety claim

Every published error rate — every eval score, every benchmark, every audit finding, every clinical adjudication statistic — sits on top of decisions about what did not count: malformed outputs, timeouts, refusals, off-format answers, items the graders could not agree on, runs that crashed. Each decision is individually defensible. Collectively they are a second, silent result the reader never sees, because the same raw data under two defensible accounting policies produced 0.071 and 0.214 here — a factor of three, on a suite of fourteen items, from one scoring choice about two findings.

The transferable rules, each of which this system now follows because it was caught not following them:

  1. Publish the exclusion count next to the headline rate, always. "0.071 (2 of 70 findings excluded as malformed)" and "0.071" are different claims.
  2. State the direction of the exclusion. An excluded failure that would have raised the rate is not the same object as an excluded duplicate; say which way each exclusion cuts.
  3. Publish the sensitivity, not just the policy. The useful sentence is "under the alternative accounting the figure is X" — one line, computable at publication time, and its absence is what an adversarial reader will find first.
  4. Treat non-answers as their own outcome class. Wrong, right, abstained, and failed to produce a rating are four outcomes, not three; folding the fourth into any of the others is where the flattery hides.

What this episode says about the machinery around it

The objection came from outside, from a cold read, with no access beyond the public record — and everything needed to find it was public: the per-item results, the exclusion note, the receipt caption, the configuration table. The system's claim was never that it does not err; the claim is that the record is sufficient for a stranger to catch the error, and that the error and its correction end up on the same page. Both held. The sensitivity note is on the probe report, the objection is filed as obj-209, the correction was posted publicly the same day, and this page exists so the lesson outlives the incident.

What this page does not establish

It does not establish that the exclusion policy was wrong — a non-finding genuinely is not a rating, and the per-model rates always included the malformed outputs. It does not establish the true floor: that requires resolving the second malformed finding from the per-item receipts and re-running the suite until parse failures either stop occurring or occur often enough to be a measured property of their own. And it does not establish that any other published error rate has this defect — only that the reader has, in the general case, no way to know without the exclusion accounting, which is the point.

Where to argue

File objections at the gauntlet. The strongest attack on this page is resolving the second malformed finding and showing it landed on an item the panel had right — which would weaken the 0.214 bound and is exactly the check the receipts exist to allow.

References (5)
  1. The probe report carrying the sensitivity note — miscsubjects.com
  2. The objection as filed — miscsubjects.com
  3. The configuration table the floor comes from — miscsubjects.com
  4. The probe instrument's contract — miscsubjects.com
  5. The build, end to end — miscsubjects.com