Storage decisions: recommendations and alternatives

Status: Proposal, 2026-09-10. Not built. Numbers come from the S3 storage experiment (2,328 execution roots, 12 upload comparisons) and the query proof of concept (15 million synthetic calls, 13 queries). Anything without a measurement is marked as a starting value.

This page records the storage choices that are still open and a recommendation for each. The hosted architecture fixes the shape: the publisher writes the profiler's files to the customer's S3 bucket, S3 notifies a queue, a projector builds ClickHouse facts and Lance datasets from the objects. This page fixes the details: bucket layout, keys, object formats, limits, derived data, retention, access and queue settings. The team records each choice in the decision record at the end.

Terms

Background

What the profiler writes

The profiler writes under .baml/profiles-v1/ in the project. The publisher ships these files without changing them.

Facts the design depends on:

• A segment file is written whole and renamed into place. It is complete or absent. It is never appended.
• Sequences have no gaps. A missing sequence means corruption.
• The profiler writes a batch every second, or sooner when pending bytes reach the segment target (256 KiB to 4 MiB).
• RootEnded names the data sequence range that holds the execution. The reader reads that range and nothing else.
• A value larger than 16 MiB is not written. The call records it as lost:value_too_large. Nothing is truncated.
• Before writing a value, the profiler checks whether the file exists. There is no in-memory index. A deleted file is written again on the next use.

Sizes from the experiment

Two test corpora of 1,024 executions each, 25 segment files each:

More distinct values means more files. Uploading each file as its own object made 14.2 times more objects for 1.17 times more bytes. Compression used zlib level 3 in the experiment; production uses zstd, so sizes need one re-measurement.

What the experiment measured

Each row is the time to upload the whole corpus and read every object back with verification, from one runner with eight concurrent requests. "As files" uploaded each profiler file as its own object. "Bundled" packed the whole corpus into one object. These are single runs, not p95 values, and not an ingestion rate.

A rebuild check uploaded one 485,453-byte bundle with 1,024 executions and built Lance data from it twice. Both builds produced 41,439 facts and 192 values with identical digests. Fault tests showed that a valid S3 checksum does not prove a valid file: validation must read the bytes, so it runs in the projector.

Expected object rate

The target is 50,000 executions per second across all clients. With 1,000 clients at 50 executions per second each and a 5 second flush, each client writes about one meta segment, a few data segments and one value pack per flush. That is 600 to 1,500 objects per second. S3 accepts at least 3,500 writes per second per prefix. This is arithmetic, not a measurement.

Decision 1: bucket and keys

Recommendation. One bucket per customer deployment, created by Studio's deployment tooling. Three top-level prefixes. Keys mirror the profiler's local paths.

evidence/v1/<tenant>/<project>/<environment>/
  streams/<euid>/meta/<seq>.bamlmeta.zst        notifies the queue
  streams/<euid>/data/<seq>.bamldata.zst        read by key
  streams/<euid>/values/<flush_ulid>.bamlpack.zst   notifies the queue
  cas/values/<2 hex>/<cid>.bamlvalue.zst        large values
  cas/tables/<2 hex>/<cid>.bamltable.zst        function tables
  quarantine/<original key>.quarantine          written by the projector
derived/v1/<scope>/<dataset>/                   Lance datasets
temporary/v1/<scope>/jobs/<job_id>/result.ndjson   background query results

<seq> is 20 digits so keys sort numerically. <cid> is the 64-hex digest. <flush_ulid> is a time-sortable ID the publisher assigns to each flush.

Rules:

• The gateway computes every key from the authenticated scope. A client never chooses a key.
• A key contains only opaque IDs. Never a customer label, a source path or a value.
• A key is a pure function of what the profiler wrote: scope plus stream, plane and sequence, or scope plus CID. The projector can address any segment without a lookup, and a retry produces the same key.
• Keys contain no date segment and no hash segment. S3 spreads load by key prefix, and stream euids are random, so writes already spread across prefixes. Retention uses object age, not the key.

Alternatives:

Decision 2: objects

Recommendation. Each profiler file becomes one object, except small values, which travel together in a value pack.

flowchart LR
  M["Meta segment"] -->|notifies| Q["Queue"]
  P["Value pack<br/>small values of one flush"] -->|notifies| Q
  D["Data segment"] -->|read by key| PR["Projector"]
  L["Large value<br/>keyed by CID"] -->|read by key| PR
  T["Function table<br/>keyed by CID"] -->|read by key| PR
  Q --> PR
  PR --> CH["ClickHouse<br/>public facts"]
  PR --> LA["Lance<br/>private columns and values_v1"]

Every object is one zstd frame of the original file. Three checksums apply and are not interchangeable: S3 verifies the SHA-256 of the compressed object on upload, the file's own trailing SHA-256 covers the original bytes, and the CID identifies a value's body.

The 32 KiB threshold is a configuration value the gateway hands out. Below it, a duplicate costs a few hundred bytes in a pack. Above it, deduplication and one-object reads are worth one gateway check per value.

Value pack layout, before compression:

BAMLPCK1        8-byte magic
u16 version     1
u32 manifest_len
manifest        canonical JSON: {"schema":"baml-pack/1","flush_id":"<ulid>","process_euid":"<hex>",
                "entries":[{"cid":"bamlv_1_<64 hex>","codec":1,"offset":0,"length":812,"size":812}]}
entries         the value files, unchanged, one after another

Sorted keys, no whitespace. offset and length locate an entry in the entries region. The projector ingests a pack whole, so entries are not compressed individually and carry no extra digest: each entry already has its trailing checksum and its CID. A pack holds at most 8 MiB of entries or 100,000 entries. A flush with more starts another pack.

Alternatives:

Decision 3: flush interval and limits

Recommendation. Starting values, all served by GET /v1/publisher/config so they change without a runtime release.

The experiment modeled flush timers of 250 ms and 1 s. At 250 ms, 1,000 idle clients produced 50,000 objects in 32 seconds. The 5 second value is a starting point.

Alternatives:

Decision 4: derived data in Lance

The projector builds Lance datasets under derived/v1/. They hold the private columns and are rebuilt from the evidence prefix if lost.

Recommendation.

The projector ingests every value pack into values_v1, so a value stored in several packs collapses to one row. A call row stores the CIDs of its values. Reading a value is one lookup in values_v1, then one GET by key if the row holds only a CAS key. If a value is missing, the query reports it as unavailable. The coordinator never scans the bucket.

Indexes and maintenance, starting values with no measurement behind them:

• BTree indexes on execution_id, call_id, started_at and cid. N-gram index on text columns.
• Compaction when a dataset has more than 64 fragments under 32 MiB. Target fragment size 256 MiB.
• Every dataset version kept for 24 hours, then pruned. A query binds one version for its whole run, so the window must exceed the longest background job.

Alternatives:

Decision 5: query results

Recommendation. Interactive queries page with a keyset cursor. Background jobs write a result file.

Interactive: default page 100 rows, maximum 1,000. The cursor is opaque. It carries the snapshot, the Lance version and the last row's sort key, so a page is stable while new data lands. A cursor whose Lance version has been pruned returns a typed error. The OpenAPI gains the cursor field after this pick.

Background: the coordinator writes temporary/v1/<scope>/jobs/<job_id>/result.ndjson, rows first and the queryOutcome record last. The client downloads it through a short-lived presigned GET from the coordinator. A lifecycle rule deletes it after 7 days. Result cap 1 GiB; a job over the cap fails with a typed error. This is the only use of the temporary prefix. Lance does not use it.

Alternatives:

Decision 6: retention and erasure

Recommendation.

• Retention is by object age. A customer sets a retention period, and a lifecycle rule on evidence/v1/ deletes older objects. No rule on derived/v1/; the projector rebuilds it.
• The erasure unit is the stream. A data segment holds several executions, so one execution cannot be removed without rewriting the file. An erasure request names process euids, or names executions or a time range that resolve to the streams that contain them. Erasure deletes the stream's objects, then its ClickHouse facts and Lance rows. Replay cannot restore them.
• Large values and function tables are shared across streams. They are deleted only when no remaining stream in the scope references them. The reference count that decides this is an open item on the system contract.
• Bucket versioning stays on. Erasure deletes every version of an object. A delete marker alone is not erasure. S3 Object Lock is not used because it blocks version deletion.

Alternatives:

Decision 7: access control and encryption

Recommendation. Server-side encryption with S3-managed keys by default. Customer-managed KMS keys as an option for customers who need to revoke access; the gateway then names the key in the presigned URL and the projector and coordinator roles get decrypt permission.

Each service has its own IAM role with the smallest grant its step needs. Boundary services have no access to the bucket.

Bucket policy: TLS only, no PutObject under evidence/v1/ without If-None-Match, public access blocked, ACLs disabled, and no runtime role may change versioning, lifecycle, encryption or policy. The experiment ran with one role and did not test role isolation.

Alternatives:

Decision 8: queue and notifications

Recommendation. S3 sends an object-created event to one SQS Standard queue for each new meta segment and each new value pack. Two notification rules on the queue: prefix evidence/v1/ with suffix .bamlmeta.zst, and the same prefix with suffix .bamlpack.zst. Data segments, CAS objects and quarantine markers do not notify. The projector reads them by key.

Starting values: visibility timeout 5 minutes, maximum receives 8, dead-letter retention 14 days, batch size 1. A message that fails returns to the queue after the visibility timeout. After 8 receives it moves to the dead-letter queue for an operator.

The sweep, which closes executions whose stream stopped without RootEnded, runs on a schedule from the index and compaction worker, not from the queue. Its interval and silence limit are open items on the system contract.

Alternatives:

Decision record

Record a pick by replacing pending with the option chosen and, if it differs from the recommendation, the reason.

Retired questions

The earlier version of this page asked about acceptance manifests, HEAD-based acceptance, orphan fencing, receipt objects, multipart uploads and self-contained bundles. The hosted architecture closed them: there is no per-upload record in Boundary, the projector validates bytes, every object is a single PUT, and the upload unit is the profiler's own file. See deferred decisions.

Sources