The system
How the system worksThe DirectoryDispatchThe LedgerAI modelsAI agentsYour installationThe appThe research libraryWhat changes for your businessUse it from your softwareEverything it can do
Messaging
iMessageWhatsApp
Ads and leads
AdsCreativeLeads
Devices
Devices
Industries
RealtorsRestaurantsHome services
Explore
How it worksPermissions and limitsCompareSet upQuestionsText me

We tightened one writing rule until the system produced 121 identical emails — and every one passed every check

The most expensive failure this build's outreach system has produced was not a rule being broken. It was a rule being obeyed. A personalisation rule existed for a good reason: openers that assert things about a recipient's website which

The failure, plainly

The most expensive failure this build's outreach system has produced was not a rule being broken. It was a rule being obeyed.

A personalisation rule existed for a good reason: openers that assert things about a recipient's website which are not verifiably on that website are the signature of automated mail, so the rule required every opening observation to be grounded in what the target site actually contained. Each time a draft leaned on a thin or generic observation, the rule was tightened. Each tightening was individually correct. The sequence of tightenings banned, one by one, every category of observation the target sites actually contained — until exactly one legal opener remained.

One hundred and twenty-one drafts then converged on that opener, under the same four-word subject line. Every one of them passed every validator. Banned-phrase checks, subject-line contract, register rules, claim-class limits — all green, 121 times. The corpus was perfectly compliant and perfectly interchangeable, and interchangeable mail is unwanted mail no matter how strict the rules that produced it were. None of it was sent; the collapse was caught in the stored corpus before the send gate, so the price was compute and embarrassment rather than 121 strangers' attention. But the system had produced, at scale, exactly the thing the rule existed to prevent — by enforcing the rule.

Why no validator saw it

Every check in the pipeline judged one draft at a time, and each draft, taken alone, was fine: polite, grounded, within register, within claim class. The defect did not live in any draft. It lived in the relationship between drafts — a property of the corpus, invisible at the only granularity the validators possessed. This is the general blind spot of per-item validation, and it is worth stating as a law because it recurs everywhere rule systems are used to govern generation:

A property can be perfect in every instance and catastrophic in aggregate, and a per-instance validator cannot see aggregate properties by construction.

Tightening per-item rules does not fix an aggregate defect. It caused this one. Each tightening shrank the space of legal drafts; a generator squeezed into a small space produces outputs that cluster; the tightest possible rule set produces identical output with a perfect compliance record. Strictness and distinctness are different properties, and past a point they trade against each other.

The detector: hash the residue

The fix is structural, and it is the useful part of this page.

A draft's shape is what remains after removing everything that is supposed to vary: the personalised opener, the catalog block, every URL and every number. What is left is the skeleton the generator actually built — transitions, framing, argument order, the ask. That residue is hashed. Two drafts written under the same effective rules produce the same hash, however different their names and links look at a glance.

Clustering the stored corpus on that hash collapses a pile of near-identical bodies into the handful of generations the copy has actually been through. Each cluster is one shape; the count of distinct businesses inside one shape is the collapse measurement — 121 businesses in one shape was this failure's number. The detector has three properties the per-item validators lacked:

  • It is aggregate by construction. It cannot be passed one draft at a time, because it does not evaluate drafts; it evaluates the corpus.
  • It needs no model and no judgment. Strip, hash, count. There is nothing to argue with and nothing to drift.
  • It measures the thing the recipient experiences. A recipient who receives interchangeable mail does not care which rules produced it; the hash count is the interchangeability, made numeric.

The regime around it: every change to the drafting rules is stored verbatim with its timestamp, and the clustering is re-run after each change — because the failure mode is a consequence of rule changes, the monitor is keyed to rule changes. A rule system that cannot see its own outputs converge will converge again.

The general lesson, because this is not about email

Substitute any generator governed by per-item rules and the anatomy holds:

  • Code review checklists. Every function passes the checklist; the codebase converges on one blessed pattern applied where it fits and where it does not. The checklist cannot see it.
  • Content policy. Every article individually compliant; the corpus converges on the one framing the policy left legal. Readers experience a site that says one thing sixty ways.
  • Model evaluations. Every output individually scored safe or on-format; the model converges on the narrow band the rubric rewards. The rubric is the personalisation rule, the mode collapse is the 121 drafts, and per-sample evaluation cannot detect it — only a distributional measurement over the output corpus can.

In each case the honest metric is the same move as the shape hash: define what is supposed to vary, remove it, and measure how much identity remains. If the residue clusters, the rules have collapsed the space, and the fix is to relax or restructure a rule — not tighten one, which is the reflex, and which digs.

What this failure bought

The tightened rule was replaced rather than tightened further: the current outreach law requires one specific observation that could fit no other recipient — a requirement about information content, which cannot converge, instead of a requirement about permitted categories, which did. The shape-hash clustering stands as a permanent gate. And the failure is recorded here at full length, under this build's standing rule that a failure published where it happened is the only form a successor model can learn from — a memory that deletes its own errors teaches its successor to repeat them.

What this page does not establish

One failure, one pipeline, one detector that caught it in the stored corpus rather than in flight. The shape hash as specified here is deliberately crude — exact hashing of stripped residue finds identical skeletons, not merely similar ones, so it underestimates collapse; a softer similarity measure would find more and require judgment this version avoids on purpose. And the claim is not that per-item validation is worthless — every check in the pipeline still runs — only that it is categorically unable to see the failure class described here, and that anyone running rule-governed generation at volume without a distributional monitor is running this failure right now, undetected, with a perfect compliance record.

Where to argue

File objections at the gauntlet. The pipeline this happened in is documented, gates and all, at outreach-machinery.

References (3)
  1. The outreach machinery, documented end to end — miscsubjects.com
  2. The build, end to end — miscsubjects.com
  3. The outreach copy law the fix now lives under — miscsubjects.com