{ "schema": "http://google-preview.miscsubjects.com/schemas/jarticle.json", "status": "published", "slug": "cloudflare-os-xl-02-ledger-as-a-table", "title": "Cloudflare OS XL part 2: turning the ledger into a queryable table", "kind": "systems", "summary": "Cloudflare OS XL part 2: turning the ledger into a queryable table", "created": "2026-08-06T03:24:45.900Z", "body": [ "*Part 2 of [Cloudflare OS XL](/a/cloudflare-os-xl), an inventory of the Cloudflare platform this build does not have installed.*\n\nThe central claim of this build is that nothing is ever overwritten and every action appends a hash-chained audit row. That claim is true. The rows exist, in a D1 database called `loop-shared-events` and in R2 as receipt files.\n\nThen someone asks a question of it — how many outbound sends went to a domain whose MX record failed, per week, since May — and the answer is produced by pulling files and counting them in a script. The ledger is a record. It is not yet a table anybody can query.\n\nFive products close the distance between those two things.\n\n## Pipelines\n\nPipelines is Cloudflare's streaming ingest: data arrives over HTTP or from a Worker binding, is transformed with SQL, and is delivered to R2 as Apache Iceberg tables or as Parquet and JSON files. It is in open beta.\n\nToday, every ledger append is a D1 `INSERT` executed inside the request that caused it. That has three costs. It puts a write in the hot path of the thing being recorded. It makes the ledger's throughput a function of D1's write throughput. And it produces rows, not columns — which is why analytical questions are answered by export-and-count.\n\nWith a pipeline, the Worker writes an event to a binding and returns. The pipeline batches, transforms and lands it in R2 in a columnar format. The record is still append-only and still hash-chained; it is simply stored as something a query engine can read.\n\nThe natural first candidates here are the three highest-volume event streams: agent turns, tool invocations, and outbound send receipts.\n\n**Verdict: install, for agent turns first.** It is beta, so it belongs on the stream where a gap would be survivable, not on the audit chain.\n\n## R2 Data Catalog and R2 SQL\n\nR2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket. R2 SQL is a distributed SQL engine that queries it. Together they are the reason the previous section says \"Iceberg\" rather than \"Parquet files in a bucket\": Iceberg gives the pile of files a schema, a snapshot history and a table identity, and R2 SQL means you do not have to bring your own engine to read it.\n\nEnabling the catalog on an existing bucket is one command.\n\n```\nwrangler r2 bucket catalog enable miscsubjects-ledger\n```\n\nWhat this changes for this build is the nature of an audit. The audit chain is the build's trust mechanism; it is what makes the claim \"nothing is ever overwritten\" checkable rather than asserted. But a trust mechanism that can only be verified by a bespoke script is verified by whoever wrote the script. A ledger as an Iceberg table can be queried by anyone with the credential, including a model, including an outside auditor, with a plain `SELECT`.\n\nThere is a second, quieter benefit. Time-travel is a property of Iceberg, not something this build would have to implement: the table can be read as of a snapshot. \"What did the ledger say on 3 August\" stops being a question about backups.\n\n**Verdict: install.** Low cost, and it converts an existing asset into a queryable one without moving it out of R2.\n\n## R2 event notifications\n\nAn object lands in R2 and a message appears on a queue. That is the whole feature, and it is missing from a build that has three queues already.\n\nRight now, assets get processed because a cron woke up and looked. Generated hero images, ArcAds output, absorbed repositories, uploaded references — each of those arrives in a bucket and then waits for a scheduled sweep to notice. The sweep runs every minute, which is fast enough to feel instant and is still the wrong mechanism: it polls whether or not anything happened, and it cannot tell you *why* it processed something.\n\n```\nwrangler r2 bucket notification create miscsubjects-store --event-type object-create --queue loop-tasks\n```\n\nWith that, the arrival of the object *is* the trigger. The queue message carries the bucket, the key and the event type, so the consumer knows exactly what changed rather than diffing a listing.\n\n**Verdict: install.** It is one command per bucket and it deletes polling code.\n\n## Analytics Engine\n\nAnalytics Engine accepts unlimited-cardinality analytics written from a Worker and queried with SQL. Writes are non-blocking and effectively free; you get one dataset binding and you write data points with blobs, doubles and an index.\n\n```toml\n[[analytics_engine_datasets]]\nbinding = \"METRICS\"\ndataset = \"loop_metrics\"\n```\n\n```js\nenv.METRICS.writeDataPoint({\n blobs: [toolName, modelId, agentName, outcome],\n doubles: [latencyMs, tokensIn, tokensOut, costUsd],\n indexes: [agentName],\n});\n```\n\nThis build already tries to answer cost and latency questions — there is a `COST_REPORT` row, a governor, and per-model accounting. Those work by reading the ledger back and aggregating it, which means the cost of asking a cost question scales with the size of the ledger.\n\nAnalytics Engine is the correct tool for that specific class of question because it is designed for high-cardinality dimensions. Per-tool, per-model, per-agent, per-outcome, forever, at a write cost that does not compete with the request. The ledger keeps being the record of what happened; Analytics Engine becomes the record of how much it cost and how long it took.\n\nThe one constraint worth knowing before adopting it: it is a metrics store, not an event store. Data points are sampled at high volume and are not the audit trail. Do not put anything in it that has to be exact.\n\n**Verdict: install.** It answers the cost question this build keeps asking, and it does not compete with the ledger for that role.\n\n## What this part does not recommend\n\n**Do not move the audit chain off D1.** The hash chain's value is that each row commits to the previous one at write time, inside a transaction, in the same request that performed the act. Streaming it through a batching pipeline first would put a gap between the act and the commitment, and the gap is exactly what the chain exists to close. Pipelines belongs on the high-volume observational streams. The chain stays where it is.\n\n## Verdicts\n\n| Product | What it replaces here | Verdict |\n| --- | --- | --- |\n| Pipelines | Per-row D1 inserts in the hot path for high-volume streams | **install** — agent turns first |\n| R2 Data Catalog | A ledger auditable only by a bespoke script | **install** |\n| R2 SQL | Export-and-count in a local script | **install** — with the catalog |\n| R2 event notifications | A cron sweep that polls buckets every minute | **install** |\n| Analytics Engine | Cost and latency questions answered by re-reading the ledger | **install** |\n| Pipelines *for the audit chain* | Nothing — it would weaken it | **no** |\n\nNext: [Part 3 — running real code](/a/cloudflare-os-xl-03-running-real-code)." ], "facts": { "converted_from": "markdown", "original_updated": "2026-08-06T03:24:45.945Z", "register": null, "category": "systems" }, "hero": "https://miscsubjects.com/img/gen/arcads-gpt-image-ebfcc12b-c321-4aaf-84ed-7f1e9b5e9022.png", "model": "Opus 5 (Claude Code)", "tags": [ "cloudflare", "pipelines", "r2", "ledger", "analytics" ], "sources": [ { "id": "s-pipelines", "type": "documentation", "url": "https://developers.cloudflare.com/pipelines/", "title": "Cloudflare Pipelines documentation", "quote": "Ingest, transform, and deliver streaming data to R2 as Apache Iceberg tables or Parquet and JSON files.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "genesis", "hash": "d83bf6dd90d89a92ef0d30c2bedd16c55cf99b0a54cbe415b6a962c3bb825be9", "quote_sha256": "35e987451b6676eb6a041f1cbe0a0247230654202ba44f22b785ad315afc8274" }, { "id": "s-catalog", "type": "documentation", "url": "https://developers.cloudflare.com/r2/data-catalog/", "title": "R2 Data Catalog documentation", "quote": "A managed Apache Iceberg data catalog built directly into R2 buckets.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "d83bf6dd90d89a92ef0d30c2bedd16c55cf99b0a54cbe415b6a962c3bb825be9", "hash": "4ecf6e4b6c3d67fa657d12b6811d9b84b4aaf48df1ea4ba1db8c2c8fe88f8c63", "quote_sha256": "8ed3930add3058bfa63417bc5685824b62680a3be14abe4011a22f0a2c470830" }, { "id": "s-r2sql", "type": "documentation", "url": "https://developers.cloudflare.com/r2-sql/", "title": "R2 SQL documentation", "quote": "A distributed SQL engine for R2 Data Catalog", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "4ecf6e4b6c3d67fa657d12b6811d9b84b4aaf48df1ea4ba1db8c2c8fe88f8c63", "hash": "6963f273f0782ed23ebeee6e74983f6ff18bd66f65bf9862dbf550a89fa2843e", "quote_sha256": "70b36015ecc5cb2d5e80a9122903427c2bb85abc79e523fcde0c6d1a7cd159aa" }, { "id": "s-ae", "type": "documentation", "url": "https://developers.cloudflare.com/analytics/analytics-engine/", "title": "Workers Analytics Engine documentation", "quote": "Send and query unlimited-cardinality analytics from Workers.", "accessed_at": "2026-08-06T03:10:03.976Z", "prev": "6963f273f0782ed23ebeee6e74983f6ff18bd66f65bf9862dbf550a89fa2843e", "hash": "161b10e67cc9e855b9b6e516a50ac2136b3b05b77a4f0c5767448763337f830c", "quote_sha256": "d9d669ed437c481f6f437e52f98747e0aea836f58a45c70286a7aa0d1df32afc" } ], "claims": [ { "id": "c1", "text": "The audit ledger exists as rows in D1 and receipt files in R2, and analytical questions about it are answered today by exporting files and counting them in a script.", "tier": "observational", "source_ids": [], "why_material": "A trust mechanism verifiable only by a bespoke script is verified by whoever wrote the script." }, { "id": "c2", "text": "Cloudflare Pipelines ingests streaming data and delivers it to R2 as Apache Iceberg tables or as Parquet and JSON files, which removes the per-row insert from the hot path of the request being recorded.", "tier": "definition", "source_ids": [ "s-pipelines" ], "why_material": "It changes ledger throughput from a function of D1 write capacity to a function of batching." }, { "id": "c3", "text": "R2 Data Catalog is a managed Apache Iceberg catalog built into an R2 bucket, and enabling it on the existing ledger bucket is a single wrangler command.", "tier": "definition", "source_ids": [ "s-catalog" ], "why_material": "It converts an existing asset into a queryable one without moving it." }, { "id": "c4", "text": "R2 SQL is a distributed SQL engine over R2 Data Catalog, so the audit chain becomes answerable with a plain SELECT by any holder of the credential rather than only by a script author.", "tier": "definition", "source_ids": [ "s-r2sql" ], "why_material": "External checkability is the point of an audit chain." }, { "id": "c5", "text": "R2 event notifications place a message on a queue when an object is created, which replaces the every-minute cron sweep that currently polls buckets whether or not anything arrived.", "tier": "definition", "source_ids": [], "why_material": "The build already runs three queues, so the consumer side exists." }, { "id": "c6", "text": "Analytics Engine accepts unlimited-cardinality data points written non-blocking from a Worker and queried with SQL, which fits per-tool and per-model cost accounting better than re-reading the ledger.", "tier": "definition", "source_ids": [ "s-ae" ], "why_material": "The cost of asking a cost question currently scales with the size of the ledger." }, { "id": "c7", "text": "The hash-chained audit rows should stay in D1 rather than move to Pipelines, because the chain commits to the previous row inside the same request that performed the act.", "tier": "expert", "source_ids": [ "s-pipelines" ], "why_material": "Batching would open the gap the chain exists to close." } ], "type": "article", "updated": "2026-10-08T11:31:32.720Z", "version": 1, "hash": "c3a1fbbc261fa69a517c3d7cdbc00e3b5ea5d80c2e0df727e5c907867f954d8a", "styling": { "scope": "shared", "how": "no value on the page is set by hand: each comes from this theme, and each theme value becomes the CSS variable named beside it. PATCH http://google-preview.miscsubjects.com/api/jarticles/theme changes them.", "widget_style": { "what": "every widget may carry style: {…}; these are the only keys it reads", "rotate": "degrees, a number", "offset_x": "pixels, a number", "offset_y": "pixels, a number", "pulse": "true to pulse" }, "theme": { "color": { "page_bg": "#f5f5f7", "sheet_bg": "#ffffff", "ink": "#1d1d1f", "soft": "#424245", "muted": "#6e6e73", "dim": "#86868b", "line": "#d2d2d7", "accent": "#0071e3", "accent_soft": "rgba(0,113,227,0.08)", "link": "#0066cc", "selection": "rgba(0,113,227,0.18)", "widget_bg": "#f5f5f7", "widget_border": "#d2d2d7", "card_bg": "#ffffff", "code_bg": "#f5f5f7", "json_bg": "#161b19", "json_ink": "#e6e1d4", "json_key": "#7fd3c3", "json_string": "#e9c46a", "json_number": "#f4a261", "json_bool": "#c792ea", "json_null": "#928d7e", "json_punct": "#6b6759", "json_bar": "#0e1512", "kind_badge_bg": "#0e6e60", "kind_badge_ink": "#ffffff" }, "color_dark": { "page_bg": "#000000", "sheet_bg": "#1d1d1f", "ink": "#f5f5f7", "soft": "#d2d2d7", "muted": "#a1a1a6", "dim": "#86868b", "line": "#424245", "accent": "#2997ff", "accent_soft": "rgba(41,151,255,0.12)", "link": "#2997ff", "selection": "rgba(41,151,255,0.25)", "widget_bg": "#2c2c2e", "widget_border": "#424245", "card_bg": "#1d1d1f", "code_bg": "#2c2c2e", "json_bg": "#0a0d0c", "json_ink": "#e6e1d4", "json_key": "#7fd3c3", "json_string": "#e9c46a", "json_number": "#f4a261", "json_bool": "#c792ea", "json_null": "#757062", "json_punct": "#5d665f", "json_bar": "#060807", "kind_badge_bg": "#4fc1ab", "kind_badge_ink": "#0b0f0d" }, "font": { "display": "-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif", "body": "-apple-system,BlinkMacSystemFont,\"SF Pro Display\",\"SF Pro Text\",\"Helvetica Neue\",Helvetica,Arial,sans-serif", "mono": "ui-monospace,SFMono-Regular,\"SF Mono\",Menlo,Consolas,monospace", "google_fonts": "" }, "size": { "body": "17px", "small": "17px", "eye": "17px", "lead": "17px", "h1": "52px", "h2": "34px", "h3": "22px", "byline": "17px", "h1_measure": "none", "code": "17px", "json": "17px", "widget_title": "17px" }, "leading": { "body": "1.7", "head": "1.18", "lead": "1.5", "json": "1.6" }, "weight": { "display": "700", "head": "700", "body": "400", "strong": "700", "eye": "700" }, "tracking": { "eye": "0.14em", "display": "-0.03em", "body": "-0.006em", "badge": "0.08em" }, "layout": { "sheet_width": "1000px", "text_width": "880px", "gutter": "16px", "sheet_padding": "56px 64px 64px", "sheet_padding_mobile": "24px 18px 40px", "sheet_margin": "28px auto 72px", "block_gap": "1.15em", "widget_gap": "28px", "grid_gap": "14px" }, "border": { "sheet": "0px solid", "sheet_radius": "30px", "widget": "1px solid", "widget_radius": "24px", "card_radius": "24px", "badge_radius": "999px", "rule": "1px solid", "code_radius": "18px", "accent_bar": "3px" }, "shadow": { "sheet": "0 1px 1px rgba(0,0,0,.02),0 6px 26px rgba(0,0,0,.05)", "widget": "none", "card_hover": "0 10px 28px -16px rgba(28,27,23,0.35)" }, "motion": { "hover": "160ms ease" } } }, "page": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table", "json": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.json", "markdown": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.md", "theme": "http://google-preview.miscsubjects.com/article-theme.json", "calls": [ { "what": "this article as a page (no key)", "method": "GET", "url": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table", "headers": {} }, { "what": "this article as its object (no key)", "method": "GET", "url": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.json", "headers": {} }, { "what": "this article as Markdown (no key)", "method": "GET", "url": "http://google-preview.miscsubjects.com/a/cloudflare-os-xl-02-ledger-as-a-table.md", "headers": {} }, { "what": "the object with its hash, drafts too", "method": "GET", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)" } }, { "what": "one piece and that piece’s hash", "method": "GET", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/at?path=facts", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)" } }, { "what": "grep: every key or value that matches, with the path to change it", "method": "GET", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/grep?q=owner", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)" } }, { "what": "change pieces (set, merge, delete, append, insert, move); if-match is the hash", "method": "PATCH", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)", "content-type": "application/json", "if-match": "" }, "body": { "ops": [ { "op": "set", "path": "facts.example", "value": 1 } ] } }, { "what": "replace the whole object", "method": "PUT", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)", "content-type": "application/json", "if-match": "" }, "body": { "slug": "cloudflare-os-xl-02-ledger-as-a-table", "title": "…" } }, { "what": "fetch every source, hash what came back and check the quote is in it", "method": "POST", "url": "http://google-preview.miscsubjects.com/api/jarticles/cloudflare-os-xl-02-ledger-as-a-table/verify", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)", "content-type": "application/json" }, "body": {} }, { "what": "an Obsidian-style command", "method": "POST", "url": "http://google-preview.miscsubjects.com/api/jarticles/cli", "headers": { "x-sheets-key": "$(cat /Users/cyrusmassoumi/.misc-sheets-key)", "content-type": "application/json" }, "body": { "command": "backlinks", "file": "cloudflare-os-xl-02-ledger-as-a-table" } } ] }