{
  "schema": "http://google-preview.miscsubjects.com/schemas/jarticle.json",
  "status": "published",
  "slug": "the-exclusion-policy-is-a-safety-claim",
  "title": "Our error rate looked three times better than it was, because of what we refused to count",
  "kind": "standard",
  "summary": "An external cold audit read this site's adjudication numbers the way an adversary should, and found an arithmetic tension nobody inside the build had published: The known-answer probe suite has fourteen items. On three of them — P05, P07,",
  "created": "2026-08-01T22:58:04.311Z",
  "body": [
    "## The finding, as it arrived\n\nAn external cold audit read this site's adjudication numbers the way an adversary should, and found an arithmetic tension nobody inside the build had published:\n\nThe known-answer probe suite has fourteen items. On three of them — P05, P07, P09 — the entire five-model panel was wrong: zero correct out of five, three separate times. Yet the published configuration table reports a five-channel floor of **one** undetected-wrong item in fourteen: 0.071, naming P07 as the sole survivor. If three items were unanimously wrong, why does only one survive every configuration?\n\nThe reconciliation was in the fine print. Two of the seventy findings were malformed — one confirmed at the receipt level as `kimi-k2.6` returning UNPARSED on P05 — and were excluded from the configuration statistics, because a non-finding is not a rating. That exclusion is a defensible scoring decision. But it has a mechanical consequence the report did not state: **a malformed finding forces the gate to escalate rather than emit.** An unparseable output on an item the panel would otherwise have answered wrongly converts an escaped error into a human referral. On at least one, and possibly two, of the three unanimously-wrong items, the assembly was rescued not by diversity, not by the gate's design, but by a model failing to produce parseable output.\n\nThe headline number — five channels drive undetected-wrong down to 0.071 — rests in part on accidental parse failures. Take the rescue away and the floor bound is 3/14 = **0.214**, roughly triple.",
    {
      "type": "source",
      "id": "s2"
    },
    "## What is confirmed and what is inference, exactly\n\nConfirmed, at the linked surfaces:\n\n- The exclusion policy exists and is stated on [the probe report](https://miscsubjects.com/a/adjudication-probe-report-eu-ai-act): 2 of 70 findings malformed, excluded from configuration statistics, retained in per-model rates.\n- P05, P07 and P09 were each 0/5 — printed per item, with the declared expected verdict and the reason it is correct.\n- `kimi-k2.6` returned UNPARSED on P05 — the receipt caption says so.\n- A malformed finding cannot be emitted; the gate's only move is escalation.\n\nNot yet resolved: **which item the second malformed finding landed on.** If it landed on P09, both rescues sit on unanimously-wrong items and the 0.214 bound binds tight. If it landed on an item the panel had right anyway, one of the three unanimous misses escaped by some other route and the accounting needs a different correction. The per-item receipts settle this and reading them is open work, stated here as open work.\n\n## Why the rescue is genuinely double-edged\n\nIt would be too quick to call this only an embarrassment. Escalating on malformed output is *correct* behaviour — a gate that emitted anyway, or guessed, would be indefensible. The assembly did, mechanically, the safe thing: faced with a channel that produced garbage on a question where every functioning channel was confidently wrong, it declined to answer. In the field, that outcome — a human looks at P05 — is strictly better than the alternative the other channels were unanimously offering.\n\nThe defect is not the behaviour. The defect is the **bookkeeping**: crediting that outcome to the assembly's measured error floor without disclosing that the mechanism was luck. A parse failure is not a safety property, because it is not reproducible on demand — the next run of P05 may parse cleanly and emit the wrong answer five-for-five. A floor propped by accident holds until the accident stops happening, which is precisely the kind of number that fails exactly when relied upon. The honest statement is now on the report: 0.071 is the floor **under the stated exclusion policy**; 0.214 is the bound under the accounting that treats rescues as escapes; a reader pricing a consequence should know which one they are holding.",
    {
      "type": "source",
      "id": "s1"
    },
    "## The general lesson: an exclusion policy is a safety claim\n\nEvery published error rate — every eval score, every benchmark, every audit finding, every clinical adjudication statistic — sits on top of decisions about what did not count: malformed outputs, timeouts, refusals, off-format answers, items the graders could not agree on, runs that crashed. Each decision is individually defensible. Collectively they are a second, silent result the reader never sees, because the same raw data under two defensible accounting policies produced 0.071 and 0.214 here — a factor of three, on a suite of fourteen items, from one scoring choice about two findings.\n\nThe transferable rules, each of which this system now follows because it was caught not following them:\n\n1. **Publish the exclusion count next to the headline rate, always.** \"0.071 (2 of 70 findings excluded as malformed)\" and \"0.071\" are different claims.\n2. **State the direction of the exclusion.** An excluded failure that would have raised the rate is not the same object as an excluded duplicate; say which way each exclusion cuts.\n3. **Publish the sensitivity, not just the policy.** The useful sentence is \"under the alternative accounting the figure is X\" — one line, computable at publication time, and its absence is what an adversarial reader will find first.\n4. **Treat non-answers as their own outcome class.** Wrong, right, abstained, and *failed to produce a rating* are four outcomes, not three; folding the fourth into any of the others is where the flattery hides.\n\n## What this episode says about the machinery around it\n\nThe objection came from outside, from a cold read, with no access beyond the public record — and everything needed to find it was public: the per-item results, the exclusion note, the receipt caption, the configuration table. The system's claim was never that it does not err; the claim is that the record is sufficient for a stranger to catch the error, and that the error and its correction end up on the same page. Both held. The sensitivity note is on the probe report, the objection is filed as [obj-209](https://miscsubjects.com/i/discourse/obj-209), the correction was posted publicly the same day, and this page exists so the lesson outlives the incident.",
    {
      "type": "source",
      "id": "s5"
    },
    "## What this page does not establish\n\nIt does not establish that the exclusion policy was wrong — a non-finding genuinely is not a rating, and the per-model rates always included the malformed outputs. It does not establish the true floor: that requires resolving the second malformed finding from the per-item receipts and re-running the suite until parse failures either stop occurring or occur often enough to be a measured property of their own. And it does not establish that any other published error rate has this defect — only that the reader has, in the general case, no way to know without the exclusion accounting, which is the point.\n\n## Where to argue\n\nFile objections at the [gauntlet](https://miscsubjects.com/a/gauntlet-log). The strongest attack on this page is resolving the second malformed finding and showing it landed on an item the panel had right — which would weaken the 0.214 bound and is exactly the check the receipts exist to allow."
  ],
  "facts": {
    "converted_from": "markdown",
    "original_updated": "2026-08-01T23:14:45.191Z",
    "register": "standard",
    "category": null
  },
  "sources": [
    {
      "id": "s1",
      "type": "live_surface",
      "title": "The probe report carrying the sensitivity note",
      "publisher": "miscsubjects.com",
      "url": "https://miscsubjects.com/a/adjudication-probe-report-eu-ai-act",
      "summary": "The 14-probe suite, the three unanimously-wrong items (P05, P07, P09), the 2-of-70 malformed exclusion, and the sensitivity paragraph this article expands, appended 2026-08-01.",
      "accessed_at": "2026-08-01T23:00",
      "claim_ids": [
        "c1",
        "c2",
        "c3",
        "c4"
      ],
      "prev": "genesis",
      "hash": "095775937b56df537899db5775bfaca1c03d862e0202edf6eee3fbabf091b487"
    },
    {
      "id": "s2",
      "type": "live_surface",
      "title": "The objection as filed",
      "publisher": "miscsubjects.com",
      "url": "https://miscsubjects.com/i/discourse/obj-209",
      "summary": "Objection 209: the 0.071 floor is sensitive to the malformed-output exclusion policy and the report did not say so. Raised by an external cold audit, 2026-08-01.",
      "accessed_at": "2026-08-01T23:00",
      "claim_ids": [
        "c1",
        "c5"
      ],
      "prev": "095775937b56df537899db5775bfaca1c03d862e0202edf6eee3fbabf091b487",
      "hash": "8a1685277c224bcd827035dc2cd75c81929c68b961630d6de56741a318a923da"
    },
    {
      "id": "s3",
      "type": "live_surface",
      "title": "The configuration table the floor comes from",
      "publisher": "miscsubjects.com",
      "url": "https://miscsubjects.com/a/logical-economics",
      "summary": "Sixty-four configurations over 70 findings; the channel-count price curve; the floor column that stops at 0.071.",
      "accessed_at": "2026-08-01T23:00",
      "claim_ids": [
        "c2"
      ],
      "prev": "8a1685277c224bcd827035dc2cd75c81929c68b961630d6de56741a318a923da",
      "hash": "b1f81ef9f9e2af64bdb330042425880f30f23b6425c79787236e8ded469b9bba"
    },
    {
      "id": "s4",
      "type": "live_surface",
      "title": "The probe instrument's contract",
      "publisher": "miscsubjects.com",
      "url": "https://miscsubjects.com/api/directory/ADJUDICATE_PROBE",
      "summary": "Known answers declared in advance, run through the identical adjudication path — the instrument whose accounting policy is at issue.",
      "accessed_at": "2026-08-01T23:00",
      "claim_ids": [
        "c3"
      ],
      "prev": "b1f81ef9f9e2af64bdb330042425880f30f23b6425c79787236e8ded469b9bba",
      "hash": "9eca490858b24ee04cb0705c992b7d1ba4b3c6dbd41bb356a395495d16a3b49d"
    },
    {
      "id": "s5",
      "type": "live_surface",
      "title": "The build, end to end",
      "publisher": "miscsubjects.com",
      "url": "https://miscsubjects.com/a/the-build-end-to-end",
      "summary": "The standing rule this correction operates under: the record never overstates, and corrections are published where the error happened.",
      "accessed_at": "2026-08-01T23:00",
      "claim_ids": [
        "c6"
      ],
      "prev": "9eca490858b24ee04cb0705c992b7d1ba4b3c6dbd41bb356a395495d16a3b49d",
      "hash": "2ffc930913129b5f77628b107da804562424e9de4c359b2e05424e95b3827e18"
    }
  ],
  "claims": [
    {
      "id": "c1",
      "text": "Three of the fourteen probe items — P05, P07 and P09 — were answered wrongly by all five channels, yet the published five-channel floor was one item in fourteen (0.071); the arithmetic reconciling those two facts runs through the malformed-output exclusion policy.",
      "section": "the finding",
      "tier": "demonstrated",
      "source_ids": [
        "s1",
        "s2"
      ],
      "why_material": "A floor of one is not obviously consistent with three unanimous misses, and the reconciliation was in fine print.",
      "evidence_class": "runtime_receipt"
    },
    {
      "id": "c2",
      "text": "Two of the seventy findings were malformed and excluded from the configuration statistics; a malformed finding forces the gate to escalate rather than emit, converting a would-be wrong answer into a human referral.",
      "section": "the mechanism",
      "tier": "demonstrated",
      "source_ids": [
        "s1",
        "s3"
      ],
      "why_material": "The rescue is real safety behaviour and accidental at once — the gate did its job for a reason nobody designed.",
      "evidence_class": "runtime_receipt"
    },
    {
      "id": "c3",
      "text": "The receipt caption confirms kimi-k2.6 returned UNPARSED on P05; which item the second malformed finding landed on is not yet resolved from the per-item receipts.",
      "section": "what is confirmed",
      "tier": "demonstrated",
      "source_ids": [
        "s1",
        "s4"
      ],
      "why_material": "One of the two rescues is confirmed at the receipt level; the other is inference until the receipts are read.",
      "evidence_class": "runtime_receipt"
    },
    {
      "id": "c4",
      "text": "Under an accounting that scores a parse-failure rescue on a unanimously-wrong item as an escaped error, the floor bound is 3/14 = 0.214, roughly triple the published 0.071.",
      "section": "the bound",
      "tier": "system",
      "source_ids": [
        "s1"
      ],
      "why_material": "A reader pricing a decision on 0.071 and a reader pricing it on 0.214 make different decisions.",
      "evidence_class": "owner_observation"
    },
    {
      "id": "c5",
      "text": "The objection was raised by an external cold audit, filed as objection 209, and the sensitivity was published on the probe report the same day.",
      "section": "the correction",
      "tier": "demonstrated",
      "source_ids": [
        "s2",
        "s1"
      ],
      "why_material": "The claim of this system is not that it does not err; it is that the error and the correction share a page.",
      "evidence_class": "runtime_receipt"
    },
    {
      "id": "c6",
      "text": "An exclusion policy is part of a safety claim: two accountings of the same 70 findings, both defensible, produce floors of 0.071 and 0.214, and any published rate that does not state its exclusions is quoting the flattering one silently.",
      "section": "the lesson",
      "tier": "system",
      "source_ids": [
        "s1",
        "s5"
      ],
      "why_material": "This transfers to every published error rate in every evaluation, not only this one.",
      "evidence_class": "owner_observation"
    }
  ],
  "type": "article",
  "updated": "2026-10-08T11:32:09.461Z",
  "version": 1,
  "hash": "8d27ec99c999356a2749451e49a7e03246ccc4e6833b5e4c1156c37de82f4ea3",
  "styling": {
    "scope": "shared",
    "how": "no value on the page is set by hand: each comes from this theme, and each theme value becomes the CSS variable named beside it. PATCH http://google-preview.miscsubjects.com/api/jarticles/theme changes them.",
    "widget_style": {
      "what": "every widget may carry style: {…}; these are the only keys it reads",
      "rotate": "degrees, a number",
      "offset_x": "pixels, a number",
      "offset_y": "pixels, a number",
      "pulse": "true to pulse"
    },
    "theme": {
      "color": {
        "page_bg": "#f5f5f7",
        "sheet_bg": "#ffffff",
        "ink": "#1d1d1f",
        "soft": "#424245",
        "muted": "#6e6e73",
        "dim": "#86868b",
        "line": "#d2d2d7",
        "accent": "#0071e3",
        "accent_soft": "rgba(0,113,227,0.08)",
        "link": "#0066cc",
        "selection": "rgba(0,113,227,0.18)",
        "widget_bg": "#f5f5f7",
        "widget_border": "#d2d2d7",
        "card_bg": "#ffffff",
        "code_bg": "#f5f5f7",
        "json_bg": "#161b19",
        "json_ink": "#e6e1d4",
        "json_key": "#7fd3c3",
        "json_string": "#e9c46a",
        "json_number": "#f4a261",
        "json_bool": "#c792ea",
        "json_null": "#928d7e",
        "json_punct": "#6b6759",
        "json_bar": "#0e1512",
        "kind_badge_bg": "#0e6e60",
        "kind_badge_ink": "#ffffff"
      },
      "color_dark": {
        "page_bg": "#000000",
        "sheet_bg": "#1d1d1f",
        "ink": "#f5f5f7",
        "soft": "#d2d2d7",
        "muted": "#a1a1a6",
        "dim": "#86868b",
        "line": "#424245",
        "accent": "#2997ff",
        "accent_soft": "rgba(41,151,255,0.12)",
        "link": "#2997ff",
        "selection": "rgba(41,151,255,0.25)",
        "widget_bg": "#2c2c2e",
        "widget_border": "#424245",
        "card_bg": "#1d1d1f",
        "code_bg": "#2c2c2e",
        "json_bg": "#0a0d0c",
        "json_ink": "#e6e1d4",
        "json_key": "#7fd3c3",
        "json_string": "#e9c46a",
        "json_number": "#f4a261",
        "json_bool": "#c792ea",
        "json_null": "#757062",
        "json_punct": "#5d665f",
        "json_bar": "#060807",
        "kind_badge_bg": "#4fc1ab",
        "kind_badge_ink": "#0b0f0d"
      },
      "font": {
        "display": "-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif",
        "body": "-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif",
        "mono": "ui-monospace,SFMono-Regular,\"SF Mono\",Menlo,Consolas,monospace",
        "google_fonts": ""
      },
      "size": {
        "body": "17px",
        "small": "17px",
        "eye": "17px",
        "lead": "17px",
        "h1": "52px",
        "h2": "34px",
        "h3": "22px",
        "byline": "17px",
        "h1_measure": "none",
        "code": "17px",
        "json": "17px",
        "widget_title": "17px"
      },
      "leading": {
        "body": "1.7",
        "head": "1.18",
        "lead": "1.5",
        "json": "1.6"
      },
      "weight": {
        "display": "700",
        "head": "700",
        "body": "400",
        "strong": "700",
        "eye": "700"
      },
      "tracking": {
        "eye": "0.14em",
        "display": "-0.03em",
        "body": "-0.006em",
        "badge": "0.08em"
      },
      "layout": {
        "sheet_width": "1000px",
        "text_width": "880px",
        "gutter": "16px",
        "sheet_padding": "56px 64px 64px",
        "sheet_padding_mobile": "24px 18px 40px",
        "sheet_margin": "28px auto 72px",
        "block_gap": "1.15em",
        "widget_gap": "28px",
        "grid_gap": "14px"
      },
      "border": {
        "sheet": "0px solid",
        "sheet_radius": "30px",
        "widget": "1px solid",
        "widget_radius": "24px",
        "card_radius": "24px",
        "badge_radius": "999px",
        "rule": "1px solid",
        "code_radius": "18px",
        "accent_bar": "3px"
      },
      "shadow": {
        "sheet": "0 1px 1px rgba(0,0,0,.02),0 6px 26px rgba(0,0,0,.05)",
        "widget": "none",
        "card_hover": "0 10px 28px -16px rgba(28,27,23,0.35)"
      },
      "motion": {
        "hover": "160ms ease"
      }
    }
  },
  "page": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim",
  "json": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim.json",
  "markdown": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim.md",
  "theme": "http://google-preview.miscsubjects.com/article-theme.json",
  "calls": [
    {
      "what": "this article as a page (no key)",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim",
      "headers": {}
    },
    {
      "what": "this article as its object (no key)",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim.json",
      "headers": {}
    },
    {
      "what": "this article as Markdown (no key)",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/a/the-exclusion-policy-is-a-safety-claim.md",
      "headers": {}
    },
    {
      "what": "the object with its hash, drafts too",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
      }
    },
    {
      "what": "one piece and that piece’s hash",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim/at?path=facts",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
      }
    },
    {
      "what": "grep: every key or value that matches, with the path to change it",
      "method": "GET",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim/grep?q=owner",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
      }
    },
    {
      "what": "change pieces (set, merge, delete, append, insert, move); if-match is the hash",
      "method": "PATCH",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
        "content-type": "application/json",
        "if-match": "<hash from the read>"
      },
      "body": {
        "ops": [
          {
            "op": "set",
            "path": "facts.example",
            "value": 1
          }
        ]
      }
    },
    {
      "what": "replace the whole object",
      "method": "PUT",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
        "content-type": "application/json",
        "if-match": "<hash>"
      },
      "body": {
        "slug": "the-exclusion-policy-is-a-safety-claim",
        "title": "…"
      }
    },
    {
      "what": "fetch every source, hash what came back and check the quote is in it",
      "method": "POST",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/the-exclusion-policy-is-a-safety-claim/verify",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
        "content-type": "application/json"
      },
      "body": {}
    },
    {
      "what": "an Obsidian-style command",
      "method": "POST",
      "url": "http://google-preview.miscsubjects.com/api/jarticles/cli",
      "headers": {
        "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
        "content-type": "application/json"
      },
      "body": {
        "command": "backlinks",
        "file": "the-exclusion-policy-is-a-safety-claim"
      }
    }
  ]
}