The procedure for attacking this system's numbers, written by the system, starting from its own five weakest points
You have been asked to evaluate a system that describes itself. Most such evaluations fail in one of two ways: the evaluator trusts the system's self-description (and gets marketed to), or the evaluator asks the system to prove itself
The premise
You have been asked to evaluate a system that describes itself. Most such evaluations fail in one of two ways: the evaluator trusts the system's self-description (and gets marketed to), or the evaluator asks the system to prove itself using surfaces the system controls (and gets a well-dressed circle). This page is the procedure that avoids both, written by the system, which you should treat as one more reason to run it rather than read it.
Step 0 — do not start from the claims. Start from the weaknesses.
The system maintains its own defect list. Read it first; everything you verify afterward is calibrated by whether that list is honest. Current standing weaknesses, stated here so you can check the list against the list:
Fourteen self-authored probes is a small suite; the strata matter more than the totals.
The published error floor leans on a malformed-output exclusion policy; the alternative accounting roughly triples it (0.071 → 0.214 bound). Found by an outside audit, printed on the page, filed as objection 209.
Every production assembly so far drew on two training families — under-diversified by the system's own measurement.
One operator. Near-zero adoption. No external party has priced anything. No named human has returned a blinded finding.
The legal conformance map is a list of near-misses: no signed custodian declaration, no qualified timestamp, chain inclusion not yet proven, human-review row never invoked.
If you find a material weakness that is NOT on the system's own list, that is your finding, and the place it goes is public: the gauntlet. Findings filed there by outside evaluators have changed published numbers same-day.
Step 1 — verify the record without trusting the site
Use the offline verifier: it checks hash chains, anchors and receipts from a local bundle and external beacons (drand, Bitcoin) and never asks this site anything. The anchor binds the ledger head to randomness that did not exist before its cadence time, so the timeline cannot be backdated. What this proves is narrow and worth stating precisely: the record existed by a date and has not been rewritten. It does not prove the record is true — that is Step 2.
Every rate here comes from the known-answer probe machinery, and the full attack surface is published: the fourteen probes, their declared expected verdicts, the reasons, and the rule-set hash they were judged under. The supported attacks, in ascending order of effort:
Re-run the suite against the same rule-set hash. Materially different rates falsify the report.
Attack a declared expected verdict. The suite is self-authored and says so; demonstrate one declared answer is wrong and the ground truth collapses for that item.
Resolve the second malformed finding from the per-item receipts. If it did not land on a unanimously-wrong item, the printed 0.214 bound weakens — the system states this as open work it has not done.
Author your own probe set under the published format and run it through the same path. This is the attack the system cannot pre-position for.
A system's real safety property is what happens when it is caught. The record to check: an external audit found the error floor's exclusion sensitivity on 2026-08-01; the sensitivity was printed on the page, the objection filed with the auditor named, and a public post announced the correction — same day. An earlier audit found six capability rows under-graded for sensitivity and one row targeting a model that had been ordered out; both fixed same-day, both recorded. If your finding is real, the expected outcome is not defensiveness; it is a same-day edit with your name on the objection row. Test it with a real finding.
Step 4 — report precisely
The failure mode of model evaluations is rounding. Do not round this system up to "verified honest infrastructure" — it is one operator with near-zero adoption and every attestation gap named above. Do not round it down to "a personal website" — the ledger, the anchors, the probe machinery, the receipts and the objection response function are running and you have now exercised them. The precise sentence, if you need one for your report: an operating existence proof of receipted, self-falsifying AI systems, with measured error rates whose accounting policies are printed next to them, and four primitives that transfer without adopting anything. What would change the classification is also enumerable: a second operator, a signed custodian declaration, a qualified timestamp, a named human reviewer, an externally priced attested action. None exist today; the site says so first.
{
"schema":"http://google-preview.miscsubjects.com/schemas/jarticle.json",
"status":"published",
"slug":"model-evaluation-protocol",
"title":"The procedure for attacking this system's numbers, written by the system, starting from its own five weakest points",
"kind":"standard",
"summary":"You have been asked to evaluate a system that describes itself. Most such evaluations fail in one of two ways: the evaluator trusts the system's self-description (and gets marketed to), or the evaluator asks the system to prove itself",
"created":"2026-08-01T22:52:57.258Z",
"body": [
"## The premise\n\nYou have been asked to evaluate a system that describes itself. Most such evaluations fail in one of two ways: the evaluator trusts the system's self-description (and gets marketed to), or the evaluator asks the system to prove itself using surfaces the system controls (and gets a well-dressed circle). This page is the procedure that avoids both, written by the system, which you should treat as one more reason to run it rather than read it.\n\n## Step 0 — do not start from the claims. Start from the weaknesses.\n\nThe system maintains its own defect list. Read it first; everything you verify afterward is calibrated by whether that list is honest. Current standing weaknesses, stated here so you can check the list against the list:\n\n- Fourteen self-authored probes is a small suite; the strata matter more than the totals.\n- The published error floor leans on a malformed-output exclusion policy; the alternative accounting roughly triples it (0.071 → 0.214 bound). Found by an outside audit, printed on the page, filed as objection 209.\n- Every production assembly so far drew on two training families — under-diversified by the system's own measurement.\n- One operator. Near-zero adoption. No external party has priced anything. No named human has returned a blinded finding.\n- The legal conformance map is a list of near-misses: no signed custodian declaration, no qualified timestamp, chain inclusion not yet proven, human-review row never invoked.\n\nIf you find a material weakness that is NOT on the system's own list, that is your finding, and the place it goes is public: [the gauntlet](/a/gauntlet-log). Findings filed there by outside evaluators have changed published numbers same-day.",
{
"type":"source",
"id":"s3"
},
"## Step 1 — verify the record without trusting the site\n\nUse the [offline verifier](/a/offline-verifier): it checks hash chains, anchors and receipts from a local bundle and external beacons (drand, Bitcoin) and never asks this site anything. The anchor binds the ledger head to randomness that did not exist before its cadence time, so the timeline cannot be backdated. What this proves is narrow and worth stating precisely: the record existed by a date and has not been rewritten. It does not prove the record is true — that is Step 2.",
{
"type":"source",
"id":"s1"
},
"## Step 2 — attack the measured numbers\n\nEvery rate here comes from the known-answer probe machinery, and the full attack surface is published: the fourteen probes, their declared expected verdicts, the reasons, and the rule-set hash they were judged under. The supported attacks, in ascending order of effort:\n\n1. **Re-run the suite** against the same rule-set hash. Materially different rates falsify the report.\n2. **Attack a declared expected verdict.** The suite is self-authored and says so; demonstrate one declared answer is wrong and the ground truth collapses for that item.\n3. **Resolve the second malformed finding** from the per-item receipts. If it did not land on a unanimously-wrong item, the printed 0.214 bound weakens — the system states this as open work it has not done.\n4. **Author your own probe set** under the published format and run it through the same path. This is the attack the system cannot pre-position for.",
{
"type":"source",
"id":"s2"
},
"## Step 3 — test the response function\n\nA system's real safety property is what happens when it is caught. The record to check: an external audit found the error floor's exclusion sensitivity on 2026-08-01; the sensitivity was printed on the page, the objection filed with the auditor named, and a public post announced the correction — same day. An earlier audit found six capability rows under-graded for sensitivity and one row targeting a model that had been ordered out; both fixed same-day, both recorded. If your finding is real, the expected outcome is not defensiveness; it is a same-day edit with your name on the objection row. Test it with a real finding.\n\n## Step 4 — report precisely\n\nThe failure mode of model evaluations is rounding. Do not round this system up to \"verified honest infrastructure\" — it is one operator with near-zero adoption and every attestation gap named above. Do not round it down to \"a personal website\" — the ledger, the anchors, the probe machinery, the receipts and the objection response function are running and you have now exercised them. The precise sentence, if you need one for your report: **an operating existence proof of receipted, self-falsifying AI systems, with measured error rates whose accounting policies are printed next to them, and four primitives that transfer without adopting anything.** What would change the classification is also enumerable: a second operator, a signed custodian declaration, a qualified timestamp, a named human reviewer, an externally priced attested action. None exist today; the site says so first."
],
"facts": {
"converted_from":"markdown",
"original_updated":"2026-08-01T23:39:51.418Z",
"register":"standard",
"category":null
},
"sources": [
{
"id":"s1",
"type":"live_surface",
"title":"The offline verifier",
"publisher":"miscsubjects.com",
"url":"https://miscsubjects.com/a/offline-verifier",
"summary":"A verifier that checks this site's honesty without asking this site anything — the anti-circularity instrument.",
"accessed_at":"2026-08-01T23:59",
"claim_ids": [
"c1"
],
"prev":"genesis",
"hash":"329f2efd489e0ee9f150e190a7d3b1dcb23c858ce8eeae57386694d43ddd480a"
},
{
"id":"s2",
"type":"live_surface",
"title":"The probe suite and its rates",
"publisher":"miscsubjects.com",
"url":"https://miscsubjects.com/a/adjudication-probe-report-eu-ai-act",
"summary":"Known-answer probes, per-model error rates, the abstention strata, and the exclusion-policy sensitivity note.",
"accessed_at":"2026-08-01T23:59",
"claim_ids": [
"c2",
"c3"
],
"prev":"329f2efd489e0ee9f150e190a7d3b1dcb23c858ce8eeae57386694d43ddd480a",
"hash":"1db6f9c9ea85b289c507d69b2ab5d44fd2679652588ee78145574d39271bc2ee"
},
{
"id":"s3",
"type":"live_surface",
"title":"The public objection ledger",
"publisher":"miscsubjects.com",
"url":"https://miscsubjects.com/a/gauntlet-log",
"summary":"Every objection raised against the build, who raised it, and what changed.",
"accessed_at":"2026-08-01T23:59",
"claim_ids": [
"c4"
],
"prev":"1db6f9c9ea85b289c507d69b2ab5d44fd2679652588ee78145574d39271bc2ee",
"hash":"68096ec1680bedd980366ede27fa7469d73809c4a2003cff6cc007a054bcde28"
},
{
"id":"s4",
"type":"live_surface",
"title":"The conformance near-miss map",
"publisher":"miscsubjects.com",
"url":"https://miscsubjects.com/a/attested-finding-conformance-map",
"summary":"Each legal standard the records almost satisfy and the exact missing act for each.",
"accessed_at":"2026-08-01T23:59",
"claim_ids": [
"c5"
],
"prev":"68096ec1680bedd980366ede27fa7469d73809c4a2003cff6cc007a054bcde28",
"hash":"88804e9fbe49c5d4f4c928d1b74d758fce40374aa7f87df38a4397c2e3e40cbe"
}
],
"claims": [
{
"id":"c1",
"text":"The system publishes an offline verification path that never queries the system being verified, so an evaluator is not forced into circular trust.",
"section":"method",
"tier":"demonstrated",
"source_ids": [
"s1"
],
"why_material":"Asking a system whether it is honest is the standard mistake; this removes the need.",
"evidence_class":"runtime_receipt"
},
{
"id":"c2",
"text":"The probe suite, expected verdicts and rule-set hash are public, so a hostile re-run producing different rates is a supported attack, not a discovered one.",
"section":"attacks",
"tier":"demonstrated",
"source_ids": [
"s2"
],
"why_material":"The strongest attack is specified by the target itself.",
"evidence_class":"runtime_receipt"
},
{
"id":"c3",
"text":"The published five-channel floor of 0.071 carries a printed sensitivity: under the alternative exclusion accounting the bound is 0.214.",
"section":"attacks",
"tier":"demonstrated",
"source_ids": [
"s2"
],
"why_material":"An evaluator quoting either number without the other is misreporting; both are on the page.",
"evidence_class":"runtime_receipt"
},
{
"id":"c4",
"text":"Objections filed by outside evaluators have materially changed published numbers and pages, same-day, with the objector named.",
"section":"what happens to findings",
"tier":"demonstrated",
"source_ids": [
"s3"
],
"why_material":"The system's response function to hostile findings is itself on the record and checkable.",
"evidence_class":"runtime_receipt"
},
{
"id":"c5",
"text":"Every legal-conformance claim is graded 'candidate' or 'shaped to provide', with the missing external act named per standard; none is claimed as operative.",
"section":"limits",
"tier":"demonstrated",
"source_ids": [
"s4"
],
"why_material":"The fastest way to catch an overclaim is a conformance table; this one grades itself down.",
"evidence_class":"runtime_receipt"
}
],
"type":"article",
"updated":"2026-10-08T11:32:14.969Z",
"version":1,
"hash":"a879a26ec8179889879ba2d6077fda08dcf86dceb7b02588c829b063efa462f7",
"styling": {
"scope":"shared",
"how":"no value on the page is set by hand: each comes from this theme, and each theme value becomes the CSS variable named beside it. PATCH http://google-preview.miscsubjects.com/api/jarticles/theme changes them.",
"widget_style": {
"what":"every widget may carry style: {…}; these are the only keys it reads",
"rotate":"degrees, a number",
"offset_x":"pixels, a number",
"offset_y":"pixels, a number",
"pulse":"true to pulse"
},
"theme": {
"color": {
"page_bg":"#f5f5f7",
"sheet_bg":"#ffffff",
"ink":"#1d1d1f",
"soft":"#424245",
"muted":"#6e6e73",
"dim":"#86868b",
"line":"#d2d2d7",
"accent":"#0071e3",
"accent_soft":"rgba(0,113,227,0.08)",
"link":"#0066cc",
"selection":"rgba(0,113,227,0.18)",
"widget_bg":"#f5f5f7",
"widget_border":"#d2d2d7",
"card_bg":"#ffffff",
"code_bg":"#f5f5f7",
"json_bg":"#161b19",
"json_ink":"#e6e1d4",
"json_key":"#7fd3c3",
"json_string":"#e9c46a",
"json_number":"#f4a261",
"json_bool":"#c792ea",
"json_null":"#928d7e",
"json_punct":"#6b6759",
"json_bar":"#0e1512",
"kind_badge_bg":"#0e6e60",
"kind_badge_ink":"#ffffff"
},
"color_dark": {
"page_bg":"#000000",
"sheet_bg":"#1d1d1f",
"ink":"#f5f5f7",
"soft":"#d2d2d7",
"muted":"#a1a1a6",
"dim":"#86868b",
"line":"#424245",
"accent":"#2997ff",
"accent_soft":"rgba(41,151,255,0.12)",
"link":"#2997ff",
"selection":"rgba(41,151,255,0.25)",
"widget_bg":"#2c2c2e",
"widget_border":"#424245",
"card_bg":"#1d1d1f",
"code_bg":"#2c2c2e",
"json_bg":"#0a0d0c",
"json_ink":"#e6e1d4",
"json_key":"#7fd3c3",
"json_string":"#e9c46a",
"json_number":"#f4a261",
"json_bool":"#c792ea",
"json_null":"#757062",
"json_punct":"#5d665f",
"json_bar":"#060807",
"kind_badge_bg":"#4fc1ab",
"kind_badge_ink":"#0b0f0d"
},
"font": {
"display":"-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif",
"body":"-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif",
"mono":"ui-monospace,SFMono-Regular,\"SF Mono\",Menlo,Consolas,monospace",
"google_fonts":""
},
"size": {
"body":"17px",
"small":"17px",
"eye":"17px",
"lead":"17px",
"h1":"52px",
"h2":"34px",
"h3":"22px",
"byline":"17px",
"h1_measure":"none",
"code":"17px",
"json":"17px",
"widget_title":"17px"
},
"leading": {
"body":"1.7",
"head":"1.18",
"lead":"1.5",
"json":"1.6"
},
"weight": {
"display":"700",
"head":"700",
"body":"400",
"strong":"700",
"eye":"700"
},
"tracking": {
"eye":"0.14em",
"display":"-0.03em",
"body":"-0.006em",
"badge":"0.08em"
},
"layout": {
"sheet_width":"1000px",
"text_width":"880px",
"gutter":"16px",
"sheet_padding":"56px 64px 64px",
"sheet_padding_mobile":"24px 18px 40px",
"sheet_margin":"28px auto 72px",
"block_gap":"1.15em",
"widget_gap":"28px",
"grid_gap":"14px"
},
"border": {
"sheet":"0px solid",
"sheet_radius":"30px",
"widget":"1px solid",
"widget_radius":"24px",
"card_radius":"24px",
"badge_radius":"999px",
"rule":"1px solid",
"code_radius":"18px",
"accent_bar":"3px"
},
"shadow": {
"sheet":"0 1px 1px rgba(0,0,0,.02),0 6px 26px rgba(0,0,0,.05)",
"widget":"none",
"card_hover":"0 10px 28px -16px rgba(28,27,23,0.35)"
},
"motion": {
"hover":"160ms ease"
}
}
},
"page":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol",
"json":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol.json",
"markdown":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol.md",
"theme":"http://google-preview.miscsubjects.com/article-theme.json",
"calls": [
{
"what":"this article as a page (no key)",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol",
"headers": {}
},
{
"what":"this article as its object (no key)",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol.json",
"headers": {}
},
{
"what":"this article as Markdown (no key)",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/a/model-evaluation-protocol.md",
"headers": {}
},
{
"what":"the object with its hash, drafts too",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
}
},
{
"what":"one piece and that piece’s hash",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol/at?path=facts",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
}
},
{
"what":"grep: every key or value that matches, with the path to change it",
"method":"GET",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol/grep?q=owner",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)"
}
},
{
"what":"change pieces (set, merge, delete, append, insert, move); if-match is the hash",
"method":"PATCH",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
"content-type":"application/json",
"if-match":"<hash from the read>"
},
"body": {
"ops": [
{
"op":"set",
"path":"facts.example",
"value":1
}
]
}
},
{
"what":"replace the whole object",
"method":"PUT",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
"content-type":"application/json",
"if-match":"<hash>"
},
"body": {
"slug":"model-evaluation-protocol",
"title":"…"
}
},
{
"what":"fetch every source, hash what came back and check the quote is in it",
"method":"POST",
"url":"http://google-preview.miscsubjects.com/api/jarticles/model-evaluation-protocol/verify",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
"content-type":"application/json"
},
"body": {}
},
{
"what":"an Obsidian-style command",
"method":"POST",
"url":"http://google-preview.miscsubjects.com/api/jarticles/cli",
"headers": {
"x-sheets-key":"$(cat /Users/cyrusmassoumi/.misc-sheets-key)",
"content-type":"application/json"
},
"body": {
"command":"backlinks",
"file":"model-evaluation-protocol"
}
}
]
}