# Deferred decisions: receipts, acknowledgments and the commit ledger

**Status:** Proposal, 2026-09-09. Extends the 2026-09-04 deferral of signed receipts to the whole per-upload acceptance path. The removal rests on the S3 storage experiment (2,328 execution roots, 12 upload comparisons) and the query proof of concept (15 million synthetic calls, 13 queries). The hosted system is not built. Revised 2026-09-10 for the publisher design.

The earlier design gave every upload a record inside Boundary. The publisher finished an upload by calling a commit endpoint. Boundary wrote the commit to a ledger, queued projection work in an outbox, and answered with an acknowledgment. A reconciler service watched for gaps. This proposal removes all of it. Boundary keeps no per-upload state. This page is the only place in the package that describes the retired mechanisms. It records what each one did, what replaces it, why the removal is safe, and what would bring any of it back.

## What this proposal does instead

The publisher is the background task inside the customer's application that ships profiler data to the cloud. In this proposal its success signal is the S3 PUT itself. The gateway returns a presigned URL bound to one object key, one size and one checksum. The URL is create-only. S3 verifies the checksum on PUT and stores the object durably. When the PUT returns 200, the upload is complete. The publisher releases the shipped files in its local store and never waits for anything else.

Work delivery is an S3 object-created notification into an SQS Standard queue. A dead-letter queue holds messages that fail too many times. The projector receives a message that names one meta segment or one value pack, reads that object, checks the file checksums and every internal reference, then projects. A value pack's entries go into the values dataset (`values_v1`), the Lance dataset that holds value bodies keyed by CID. Validation happens where the bytes are.

The ingest flow, with the same step numbers as the [architecture page](baml-studio-final-architecture.html):

1. The profiler writes closed segment files and value files into the local store. A file is either complete or absent.
2. Every flush interval, the publisher collects every complete file not yet shipped: meta segments, data segments, small values into one value pack, and large values and function tables as single objects.
3. The publisher compresses each object with zstd and calls the gateway once per flush with the list of objects, their sizes and their checksums.
4. The gateway validates the publisher token, computes each object key from the authenticated scope, checks S3 for each key, and answers per object: a create-only presigned PUT URL when the object is missing, or `stored` when it is already there with the same checksum.
5. The publisher PUTs the function tables, large values and value packs first, then the data segments in sequence order, then the meta segments.
6. After the meta segments are stored, the publisher advances its cursor, marks the shipped values, and releases the shipped segments to the local store. The flush is complete. Nothing waits for Boundary.
7. S3 sends an object-created notification to the projection queue for each meta segment and each value pack. Data segments and CAS objects do not notify.
8. A projector that receives a value pack notification ingests its entries into the values dataset in Lance, keyed by CID, so duplicates collapse.
9. A projector that receives a meta segment notification decodes it. For each `RootEnded` record it reads that execution's data segments by key, its function table by CID, and its values from the values dataset or from the CAS prefix.
10. The projector builds deterministic public fact batches and sends them to the Boundary projection ingest API, which writes them to ClickHouse and answers only when they are visible.
11. The projector commits the execution's private rows as a new Lance dataset version and deletes the queue message. A failure at any step leaves the message to be redelivered; after the retry limit it moves to the dead-letter queue.
12. In the background, the sweep closes executions whose stream stopped without `RootEnded` as `abandoned`, and the index and compaction worker updates indexes and merges small fragments.

A retry lists the same objects in a new authorize call, with the same sizes and checksums. The gateway checks whether each key already exists. A matching checksum returns the state `stored` for that object, which is success. A different checksum returns the state `conflict`. For a segment or a value pack that is a local bug: the publisher reports it and never overwrites. For a large value or a function table the publisher treats it as stored, because the CID already proves the content.

The replay tool is an operator command. It lists the meta objects and value packs in the evidence prefix and compares the root ids they carry with the root ids projected into ClickHouse and Lance. It enqueues a synthetic notification for each meta object or value pack whose facts are missing. It rebuilds after loss, repairs a missed notification and replays the dead-letter queue after a fix. It is not a running service.

```mermaid
sequenceDiagram
  participant U as Publisher
  participant G as Customer gateway
  participant S3 as Customer S3
  participant Q as SQS queue
  participant P as Projector

  U->>G: Authorize flush with the list of objects, sizes and checksums
  G-->>U: Per object, a create-only presigned PUT URL or stored
  U->>S3: PUT tables, large values and packs, then data segments, then meta segments
  S3-->>U: 200 per object, checksum verified
  Note over U: Flush complete, local store released
  S3->>Q: Object-created notification per meta segment and value pack
  Q->>P: Deliver message
  P->>S3: Read the meta segment or pack, then data segments and CAS objects by key
  P->>P: Project public facts and private rows
  P->>Q: Delete message
```

There is no commit step. Nothing in the diagram writes to Boundary before the projector sends facts.

## What was retired and what replaces it

| Mechanism | What it did | What replaces it | Why it is safe to remove |
| --- | --- | --- | --- |
| Signed receipts | A signed Boundary artifact proving acceptance of one upload. Deferred 2026-09-04. | Nothing. The S3 object and its checksum are the evidence. | No customer requirement for independently verifiable proof exists. Signing did not improve durability. |
| Ordinary commit acknowledgments | An unsigned response naming the commit, the uploaded object and its manifest digest. | The 200 response from the checksum-verified S3 PUT. | S3 already returns a durable, checksum-bound success. A second acknowledgment added latency, not safety. |
| Publisher commit call | `POST /v1/uploads/{authorization_id}/commit` after the PUT. | None. The PUT ends the upload. | The call only told Boundary what S3 already knew. A lost commit response needed its own retry and recovery path. |
| Boundary commit ledger | A metastore table of accepted commits, one row per upload. | The evidence prefix, listed by meta objects. The replay tool runs that listing. | The bucket is the canonical list of meta segments, and each meta segment names the executions it closes. A second list can drift from it and must be reconciled. |
| Projection outbox | Pending projection work written in the same transaction as the commit. | The SQS queue fed by S3 notifications. | SQS gives at-least-once delivery with a visible failure lane. Projection is duplicate-safe, so the outbox's exactly-once guarantee is not needed. |
| Dispatch recording | Boundary endpoints that marked a commit dispatched and then complete. | The queue message lifecycle: received, deleted or dead-lettered. | The state lived in two places, the ledger and the queue. One place is enough. |
| Reconciler service | A running service that found accepted commits with no completed projection. | The replay tool, run by an operator. | The gap it looked for is the set of meta objects in S3 whose executions have no facts. Listing the bucket answers that on demand. |
| Contiguous accepted watermarks and source ranges | Per-source sequence ranges advanced only when every earlier range was accepted. Cleanup waited on them. | Stream order and the data range carried by `RootEnded`. The publisher ships data segments before meta segments and releases local files once the meta segments are stored. | The only order that matters is inside one stream, and the publisher enforces it by upload order. A `RootEnded` record names exactly the data segments its execution needs, so nothing outside that range is waited for. The publisher keeps a per-plane cursor in its local store; nothing in Boundary tracks it. |
| Queryable watermark | A per-scope high-water mark that a query bound to define freshness. | The bound ClickHouse generation and Lance version, reported in `queryOutcome`. | Freshness is reported as what the query actually read, not as a promised position in a sequence. |
| HEAD-only verification | The gateway checked object metadata before recording a commit. | The projector reads the whole meta segment, its data segments and its values before projecting. The gateway still checks whether each object exists at authorize time, but only to detect a repeated upload. | The experiment showed that HEAD proves nothing about content. See "Why". |
| Candidate hydration | Batch-reading value bodies from S3 to evaluate predicates at query time. | Private rows and n-gram indexes in Lance, read through the NVMe cache. | Lance holds typed columns and indexes, so predicates run over columns, not over fetched bodies. |
| Opaque value handles | A scope-bound reference stored in ClickHouse for each value, resolved in the data plane to read the bytes. | The call identifier and value CIDs in ClickHouse, and deterministic Lance row IDs keyed by the execution's root id, relation and partition. | The private row is found by identity, so no separate reference needs encoding, revocation or a mapping to keep. |

For a reader of the 2026-09-04 page: the acknowledgment described there and the transaction behind it are gone. So are the contiguous ranges that gated cleanup and the recovery of a lost acknowledgment by retrying the same commit. The immutable conflict rule survives in a different place. It is now the create-only PUT and the gateway's existence check at authorize time. Both run in the customer cloud with no Boundary call.

## Why

- S3 durability plus a checksum-verified, create-only PUT already gives the publisher a durable success signal. The object key and checksum identify the object forever.
- SQS Standard with a dead-letter queue already gives at-least-once work delivery with a visible failure lane. The projector is deterministic and duplicate-safe, so at-least-once is enough.
- The storage experiment showed that acceptance metadata could not prove an upload was valid.[^faults] The fault tests, run against the experiment's benchmark bundle format rather than the production objects, injected a malformed manifest with a valid checksum, a missing data sequence and a missing function table. Each was detectable only by reading content. Acceptance therefore had to read the bytes anyway. The projector already does that.
- Removing the ledger removes two metastore transactions, one endpoint pair, a dispatch state machine and a reconciler service.
- Fewer things to operate and test. Each retired piece had its own lost-response, crash and retry cases in the acceptance suite. Those cases now collapse into "the PUT succeeded or it did not" and "the message was deleted or it was not".

## What is lost

- No Boundary-side record that a specific upload was processed. Boundary cannot answer "did you accept upload X" from its own state. It can answer "do I hold facts for execution X", and the bucket answers the rest. Acceptable for this proposal because no customer has asked for the former and the latter is what queries need.
- No cross-stream ordering. Executions project in any order across streams, and a projector may receive a meta segment before the value pack it needs has been ingested. Acceptable because the publisher's upload order puts every data segment, value pack, large value and function table in the bucket before the meta segment that references them, a message whose values are not yet ingested is retried, and queries bind one snapshot, so order never changes a result.
- Freshness is best-effort until measured. The gap between PUT and queryable depends on queue depth and projector throughput. Nothing promises a bound. Acceptable because `queryOutcome` reports the generation and Lance version each query read. The older POC workflow already showed the lag. Its only measured arrival replay, 100,000 calls over 2 seconds, took 40.85 s to confirm acceptance and 214.93 s to become visible. A promised bound would have been false anyway.
- A missed notification is repaired by replay, not prevented. S3 notifications can be lost. Until an operator runs the replay tool, the meta segment or value pack sits in S3 unprojected. Acceptable because the loss is rare, the bytes are never at risk, and the repair is one listed command with no data to reconstruct.

## When to reconsider

- A customer needs independently verifiable proof that Boundary accepted a specific upload. That brings back a signed receipt, with its payload format, key custody and rotation, and client verification.
- A regulated workflow needs Boundary to attest that specific evidence was processed. That brings back a per-upload record, though not necessarily the full ledger and outbox.
- A measured notification loss rate that replay cannot cover. If missed notifications become frequent enough that operator replay is too slow, a scheduled reconciler returns.
- A need for coverage tracking across streams. Values are already shared across flushes and processes through the CAS prefix and the values dataset, and a missing value only delays one message. If a later choice on the [storage decisions page](baml-studio-s3-design.html) makes an execution depend on another stream's objects, ordering and coverage tracking return in some form.

Before any of these are adopted, define the proof or failure scenario, the recovery objective, and who owns the retry. Then check whether the bucket plus the replay tool already meet it.

## Documentation and API consequence

The architecture, the system contract and the OpenAPI document omit receipt, commit, dispatch and watermark fields. The OpenAPI version is `0.3.0-proposal`. The removed endpoints are `/evidence/commits`, `/evidence/commits/{commit_id}`, `/projection-dispatches/pending`, `/projection-dispatches/{commit_id}`, `/projections/commits/{commit_id}/complete` and `/v1/uploads/{authorization_id}/commit`. No field is reserved for any of them. A future receipt or attestation feature needs its own contract revision.

The acceptance tests that exercised the old path are replaced. Lost acknowledgment, commit retry, contiguous gap and dispatch completion are gone. In their place: retry after a lost PUT response, duplicate notification, projector crash mid-execution, malformed meta segment or value pack to the dead-letter queue, and missed notification recovered by replay.

[^faults]: Fault tests in the S3 storage experiment. See `s3-experiment/docs/BAML-STUDIO-CONTRACT-CHANGES.md` and `s3-experiment/storage-bench/integration/README.md`.