# BAML control-plane / data-plane contract

**Status:** Proposal, 2026-09-09. Revised 2026-09-10 for the publisher design. Supersedes the 2026-09-01 scale-out draft contract and the 2026-09-04 acceptance decision. It incorporates the query proof of concept (15 million synthetic calls, 13 queries) and the S3 storage experiment (2,328 execution roots, 12 upload comparisons, one accepted upload in the experiment's benchmark bundle format rebuilt into query data twice). The hosted system is not built.

Boundary services do not connect to the data plane. Data-plane workloads connect to Boundary over outbound HTTPS and send only registered non-value fields. Profiler and query clients connect to data-plane endpoints. Evidence, source material, arguments, outputs and errors stay in data-plane S3.

## Scope

Customers keep evidence and exact values in their data plane. Boundary runs shared configuration and analytical services.

- The publisher calls the data-plane gateway. The publisher is the in-process background task inside the customer's application that ships the profiler's closed files to the cloud. The profiler is the in-process library that records calls and writes those files.
- Studio running in the browser, the CLI or an agent calls the data-plane query coordinator.
- Data-plane workloads initiate outbound HTTPS requests to Boundary.
- Boundary control-plane servers cannot initiate a connection into the data plane. Boundary replies on the connection the data plane opened.
- The data plane reads exact values and returns them only to the authorized query client.

This applies to customer-hosted BYOC and Boundary-hosted Alien data planes.

```text
ALLOWED

Profiler publisher ----------------------> Data-plane gateway
Studio browser, CLI or agent -----------> Data-plane query coordinator
Data-plane workloads -- outbound HTTPS -> Boundary workload API
S3 object-created event ---------------> Data-plane projection queue (SQS)

NOT ALLOWED

Boundary control-plane server -----X----> Data plane
```

### Data sent to Boundary

- Non-value configuration data: tenants, projects, environments, deployments, tokens, catalog versions and the active projection generation.
- Non-value analytical facts: execution and call identifiers, function names, revision IDs, status, timing, health counters and availability states such as `args_state`.[^fqn] Token counts are not recorded today and would need a catalog version bump.
- Object keys and CIDs are opaque identifiers Boundary may hold; value bodies never. A CID is the content identifier of one value, a SHA-256 over the codec and the body.
- Resident subqueries over those facts, and root ids in erasure requests. A resident subquery is the public part of a query plan.

### Data kept in the data plane

- Exact arguments, outputs and error values.
- Source text, source paths and snapshots.
- Meta segments, data segments, value packs, large values and function tables: the immutable objects the publisher writes to S3.
- Lance datasets with private typed columns, the values dataset (`values_v1`), and their scalar and n-gram indexes.
- Private literals and private field paths in query predicates, such as `args['invoice']['id'] = 'INV-1042'`.
- Background job result files.

Values stay in the data plane. Registered relation schemas define which non-value fields Boundary may process.

[^fqn]: Function names and revision IDs are treated as public facts in this proposal. Obfuscating or mapping them customer-side remains a possible later change and would not alter the contract shape.

## Contract scope

This document defines workflow behavior, state transitions, failure handling and ownership. The OpenAPI contract defines HTTP paths, methods, headers, bodies and status codes. Resolve conflicts before implementation.

The [Storage decisions](baml-studio-s3-design.html) page records the recommended bucket layout, the mapping from profiler files to objects, the value pack format, the Lance layout and the queue settings, with alternatives. Its examples are design inputs. They do not add fields or endpoints to this contract. The publisher guide describes the publisher's algorithm and the profiler facts it relies on.

## Identity and deployment model

`tenant_id` identifies a Boundary customer organization and is the root authorization boundary. A tenant owns projects, and a project owns environments.

PlanetScale is the configuration metastore. It is the system of record for this hierarchy, deployment assignments, tokens, catalog versions and the active projection generation. It holds no per-object and no per-flush state.

- A BYOC deployment is assigned to one tenant.
- A Boundary-hosted Alien cell may run deployments for multiple tenants. Sharing a cell does not change ownership.
- Every service validates scope against authenticated identity and PlanetScale.
- A scope supplied in a request body is not authority by itself.

## Surfaces and ownership

The contract has two HTTP surfaces and one queue message.

| Surface | Caller | Owned by | Network direction |
|---|---|---|---|
| Public data-plane API | Publisher, Studio, CLI or agent | Data plane | Client to data plane |
| Workload API | Gateway, projector or query coordinator | Control plane | Outbound HTTPS from data plane to Boundary |
| S3 notification message for meta segments and value packs | S3, or the replay tool | Data plane | S3 to SQS to projector |

The public data-plane API runs in a customer VPC or a Boundary-hosted Alien cell. Studio must reach it from the browser. A BYOC route may be public or use customer-network access such as a VPN or private-access proxy.

The control plane hosts the workload API. Data-plane workloads open every connection to it, and responses use the same connection.

The queue message is the standard S3 event notification JSON. The replay tool emits the same shape. The replay tool is an operator command that lists the meta objects and value packs under the evidence prefix, compares the root ids they carry with the projected root ids in ClickHouse and Lance, and enqueues a message for each object whose executions or values are missing.

## Authentication

The contract defines three authentication contexts:

1. A Boundary user token authorizes Studio, the CLI or an agent to call the query coordinator for one tenant, project and environment. Erasure requires an additional erasure permission on the token.
2. A publisher token authorizes the publisher to call `GET /v1/publisher/config` and `POST /v1/uploads/authorize` at the gateway for one environment. It is issued by the control API and rotated by the customer. A 401 or 403 response means the token is invalid: the publisher stops shipping, keeps its files within the disk budget, and retries the config endpoint later.
3. A data-plane workload identity uses mTLS and a short-lived workload JWT to call the Boundary workload API as a deployment and cell.

Every request carries a caller-generated `X-Baml-Request-Id` header. Mutations that create a job, an erasure or a projection erasure also carry an `Idempotency-Key` header. Repeating a request with the same key returns the same resource.

OpenAPI permission names are provisional and non-exhaustive. Define the final permission set and endpoint matrix during implementation.

## Contract invariants

- Boundary never initiates a connection into the data plane.
- Only non-values cross into Boundary. Values remain in data-plane S3 and in derived Lance datasets in the same bucket.
- Scope is derived from authenticated identity and PlanetScale, not request JSON alone.
- An object in S3 is immutable. Its key is a pure function of scope, stream, plane and sequence, or of scope and CID, and its checksum identifies it forever. A successful S3 PUT is the only upload success signal.
- A meta object's notification fires only after every data segment it references and every pack, large value and table those segments reference exist in the bucket, because the publisher uploads in that order. There are no per-upload records in Boundary.
- Notification and queue delivery are at least once. Projection is deterministic and duplicate-safe. An execution is projected exactly once per generation, keyed by its root id.
- A query binds one generation, one catalog version and one Lance version for its whole execution. Unavailable evidence is reported in `queryOutcome`, never silently dropped. A streamed result is incomplete without its final outcome record.
- ClickHouse facts, Lance datasets, indexes, the NVMe cache and queue messages are rebuildable from the bucket.

## Endpoint directory

:::component openapi-endpoint-directory
:::

## Evidence ingestion

### Storage model

Three systems answer three different questions:

| System | Question answered |
|---|---|
| Data-plane S3 evidence prefix | Does this file exist? |
| Projection queue (SQS) | Which meta segments and packs still need processing? |
| Boundary ClickHouse and data-plane Lance | What is queryable? |

S3 is the canonical evidence store and mirrors the profiler's local stream layout. The queue carries work, not state. ClickHouse and Lance are derived and rebuildable. No system outside the bucket records that an individual upload happened.

### Object kinds

A stream is one process's file sequence, identified by a 32-hex process euid. A segment is one immutable file in a stream plane, meta or data. The publisher ships five kinds of object. The `kind` value is the one used in the authorize request.

| `kind` | Key under `evidence/v1/<tenant>/<project>/<environment>/` | Identity | Notifies the queue |
|---|---|---|---|
| `meta_segment` | `streams/<process_euid>/meta/<seq>.bamlmeta.zst` | stream and sequence | yes |
| `data_segment` | `streams/<process_euid>/data/<seq>.bamldata.zst` | stream and sequence | no, read by key |
| `value_pack` | `streams/<process_euid>/values/<flush_ulid>.bamlpack.zst` | stream and flush | yes |
| `value` | `cas/values/<2 hex>/<cid>.bamlvalue.zst` | CID, codec 1 | no, read by key |
| `table` | `cas/tables/<2 hex>/<cid>.bamltable.zst` | CID, codec 2 | no, read by key |

`<seq>` is 20 decimal digits. `<cid>` is the 64-hex digest without the `bamlv_1_` prefix. A value pack holds the small values of one flush. A large value is a value at or above the large-value threshold and is stored as its own object. A function table is a CAS object with codec 2. Every object, packs included, is one zstd frame of its original bytes.

### Upload sequence

The step numbers match the ingest flow on the [architecture page](baml-studio-final-architecture.html). A flush is one publisher cycle.

```mermaid
sequenceDiagram
  participant U as Publisher
  participant G as Gateway
  participant S3 as S3
  participant Q as SQS
  participant P as Projector
  participant B as Boundary ingest API
  participant CH as ClickHouse
  participant L as Lance

  U->>U: 1-2. Profiler closes files; publisher collects the flush
  U->>G: 3. POST /v1/uploads/authorize with every object
  G->>S3: 4. HEAD each key
  G-->>U: 4. Per object: create-only presigned PUT URL, or stored
  U->>S3: 5. PUT tables, large values, packs, then data, then meta
  S3-->>U: 5. Stored durably
  U->>U: 6. Advance cursor, mark values, release segments
  S3->>Q: 7. Object-created for each meta segment and pack
  Q->>P: 8. Deliver pack message
  P->>L: 8. Ingest entries into the values dataset by CID
  Q->>P: 9. Deliver meta message
  P->>S3: 9. Read data range, table and values by key
  P->>B: 10. PUT /projection-batches/{batch_id}
  B->>CH: 10. Insert facts
  B-->>P: 10. Batch visible
  P->>L: 11. Commit new dataset version
  P->>Q: 11. Delete message
  Note over P,L: 12. Sweep closes abandoned executions; index and compaction run
```

The publisher runs a flush every `flush_interval_seconds`, earlier when pending pack bytes exceed `pack_max_entry_bytes`, and on shutdown. Nothing waits for Boundary. The flush is complete when the meta segments are stored.

### Publisher configuration

`GET /v1/publisher/config` is called at start and every five minutes, and the response is cacheable. It returns `capture_enabled`, `supported_segment_versions` (`[2]`), `supported_cas_versions` (`[1]`), `flush_interval_seconds` (5), `max_objects_per_authorize` (100), `large_value_threshold_bytes` (32 KiB), `max_object_bytes` (256 MiB), `pack_max_entry_bytes` (8 MiB), `shipped_value_retention_seconds` (3,600) and `upload_url_ttl_seconds` (900). The values in parentheses are starting values. Changing one changes publisher behavior without a runtime release. The OpenAPI contract describes every field.

### Authorize request and response

One `POST /v1/uploads/authorize` per flush lists every object the flush wants to write, at most `max_objects_per_authorize` per call. The request carries `process_euid`, `flush_id` and `objects`. Every entry carries `kind`, `size_bytes` and `checksum_sha256` of the zstd object. Segments add `sequence`, packs add `entry_count`, and values and tables add `cid` and `codec`.

The gateway validates the publisher token, computes each `key` from the authenticated scope and the entry, and checks S3 for each key. The response repeats `flush_id` and `objects` in the same order. Every entry carries `kind`, its `sequence` or `cid`, its `key` and a `state`:

| `state` | Meaning | Publisher action |
|---|---|---|
| `upload_required` | No object exists under the key | PUT to `upload_url` with every header in `required_headers` before `expires_at` |
| `stored` | An object with the same key and checksum exists | Treat the object as shipped; no URL is returned |
| `conflict` | An object with the same key and a different checksum exists | Segments and packs: report a bug and never overwrite. Values and tables: treat as stored, because the CID already proves the content |

`required_headers` carries `If-None-Match: *`, `x-amz-checksum-sha256` and `Content-Length`, so the URL is bound to that key, size and checksum. A 401 or 403 stops shipping. A 413 means one object exceeds `max_object_bytes` or the request lists more than `max_objects_per_authorize` objects; the publisher splits the request or drops the oversize object. A 429 or 503 means back off with jitter.

### Object and execution states

Each object passes through four states per plane. None of them is recorded in Boundary.

| State | Meaning | Applies to |
|---|---|---|
| written locally | The profiler closed the file. It is complete or absent, never partial | all kinds |
| shipped | The object exists in S3 with a verified checksum, and the flush that carried it has completed | all kinds |
| notified | A message for the object is in the projection queue or being processed | meta segments and value packs |
| projected | The executions the object closes are projected, or its values are in the values dataset | meta segments and value packs |

An execution has three states, computed from the stream and the query stores. It is `pending` once its `RootEnded` has been seen but the execution is not yet projected, including while a value it references is not yet ingested. Before `RootEnded` arrives the execution is not known to the cloud at all; the catalog reports it as `running` only for local reads. It is `projected` when its public fact batches are visible in ClickHouse and its private rows are committed in Lance. It is `abandoned` when its stream stopped without `RootEnded` and the sweep closed it.

The coordinator computes `private_state` from the bound Lance version. A public fact whose private rows are not in the bound Lance version carries `private_state = pending`; once they are, it carries `private_state = indexed`. Background index and compaction work does not change object or execution state.

### Idempotency

Upload retries:

- Keys are pure functions of scope, stream, plane and sequence, or of scope and CID. The publisher does not choose keys and cannot collide with itself.
- A repeat flush re-authorizes with the same objects and gets `stored` for every object that landed. A crash before the cursor advances re-ships the whole flush; duplicates are harmless. An expired URL is replaced the same way, and expiry never permits a different key or different bytes.
- The presigned URL is create-only (`If-None-Match: *`). A PUT that fails the create-only precondition with 412 means the object already exists. The publisher re-authorizes that object and expects `stored`.
- `conflict` on a segment or a pack is a local bug. The publisher reports it and never overwrites. `conflict` on a value or a table is treated as stored.
- A pack may hold a value that an earlier pack or another stream already shipped. The projector collapses duplicates by CID when it ingests the pack.

Projection batches:

- `batch_id` is deterministic from the root id, `relation` and `partition`. Fact row IDs are deterministic from the same inputs plus the call position. The generation travels in a header, so a rebuild into a new generation reuses the same batch IDs.
- A `PUT /projection-batches/{batch_id}` whose `content_digest` matches a batch already visible in that generation returns the existing visible state. A reused `batch_id` with a different `content_digest` is rejected as a conflict.
- Lance row IDs are deterministic from the root id, relation and partition. The values dataset is keyed by CID. The projector merges on the row ID, so a duplicate run produces one row set, not two.
- A duplicate notification or a projector crash mid-execution is completed by redelivery with the same IDs and the same result.

## Projection contract

The projector receives an S3 notification, not an evidence body. The message names the bucket, `object.key`, `object.size`, `object.eTag` and `object.versionId`. The key suffix selects the handling.

On a value pack (`.bamlpack.zst`): the projector decompresses the pack, reads the manifest and ingests each entry into the values dataset in Lance, keyed by CID with codec, body and locator. Duplicates collapse. No public fact leaves the data plane.

On a meta segment (`.bamlmeta.zst`): the projector decodes the segment. For each `RootEnded` record it:

1. reads meta sequence 1 of the stream by key, because `StreamStarted` is record 1 of meta sequence 1 and every timestamp in the stream is `StreamStarted.zero_unix_ns` plus a nanosecond offset; then walks back through earlier meta objects of the same stream, by key in descending sequence, to find the execution's `RootStarted` and the `EngineStarted` for its engine. Only those two records need the walk. The walk is bounded to 2,000 meta objects. A walk that exhausts the bound without finding both records is treated like a sequence gap and marks the execution `index_corrupt`;
2. reads data segments `[data_first_seq ..= data_last_seq]` of the stream by key, the range carried by `RootEnded`, and skips groups that belong to other roots;
3. reads the function table by the CID in `EngineStarted.function_table_cid`;
4. resolves each value CID from the values dataset, else from the `cas/values/` prefix by key, else reports the execution `pending` and leaves the message for a retry;
5. verifies that every key lies under the evidence prefix of its own deployment scope, that every file's trailing checksum matches, that the segment versions are supported, and that the active projection generation from Boundary is the one it will write to. An S3 checksum alone does not qualify an object; the storage experiment showed that it cannot prove complete dependencies, which here are the data range, the table and the values;
6. emits ClickHouse batches and Lance rows with deterministic IDs keyed by the root id, and deletes the message once every batch is visible and the Lance version is committed.

The projector produces two outputs per execution:

| Output | Content | Destination |
|---|---|---|
| Public fact batches | zstd Arrow IPC in the registered relation schema for the six public relations of catalog v1: `threads_v1`, `call_path_stats_v1`, `calls_v1`, `errors_v1`, `function_definitions_v1` and `health_v1` (the internal `processes_v1`, `store_files_v1` and `value_index_v1` are not public) | `PUT /projection-batches/{batch_id}` on Boundary |
| Private fragments | Private typed columns, value references, source paths and spans, and n-gram index data | Lance datasets under the derived prefix in the bucket |

The availability states are written as facts with the catalog's values. In `calls_v1`, `args_state`, `output_state` and `error_state` take exactly `available`, `not_captured`, `not_applicable` or `lost:<reason>`, with reasons such as `value_too_large` and `queue_full`; a value is whole or lost with a reason, never truncated. In `threads_v1`, `status` takes `succeeded`, `failed`, `cancelled`, `panicked`, `abandoned`, or `running` for a live stream; `value_state` takes `none`, `partial` or `complete`; and `index_state` is the meta-plane integrity state, `complete`, `index_corrupt`, `root_started_lost` or `no_root_ended`. `program_id` is random per engine today, so `call_path_id` values are not comparable across processes and `function_definitions_v1` grows by one row set per engine start until the planned content-hash program id lands.

Each batch request carries its identity in headers:

| Header | Content |
|---|---|
| `X-Baml-Root-Id` | Root id of the execution the batch was built from |
| `X-Baml-Project-Id` and `X-Baml-Environment-Id` | Scope, checked against the workload identity |
| `X-Baml-Relation` | Registered relation name |
| `X-Baml-Partition` | Partition index within the execution and relation, from 0 |
| `X-Baml-Projection-Generation` | Active generation for the scope |
| `X-Baml-Content-Digest` | SHA-256 of the uncompressed Arrow body |
| `Content-Encoding` | Always `zstd` |

Boundary accepts only the registered relation schema. The projector reads the schema for its catalog version from `GET /projection-schemas/{catalog_version}`. Value-bearing fields, source material, unknown fields and unapproved string columns are rejected.

The Lance commit is the last step. It appends the new fragments and commits a new dataset version. Two projectors committing at once produce a conflict. The loser retries the commit against the newest version. Fragments are never rewritten on retry.

Executions whose stream stops without `RootEnded` are closed by the sweep. The sweep is the scheduled job that lists the stream prefixes under the evidence prefix, finds the streams whose newest meta object is older than the silence limit, and marks their executions with a `RootStarted` and no matching `RootEnded` as `abandoned` with the facts it has. The sweep interval and the silence limit are open implementation decisions below. An interior sequence gap in a plane is corruption: the projector marks every execution whose data range crosses the gap `index_corrupt`.

An object that fails verification is quarantined. The projector sends the message straight to the dead-letter queue and writes a quarantine marker under `quarantine/<original key>.quarantine`. The object bytes are kept. Operators inspect and replay after a fix.

## Failure handling

S3, SQS, ClickHouse and Lance do not share a transaction. Determinism and redelivery repair every incomplete transition.

| Failure | Durable state | Recovery |
|---|---|---|
| Gateway unreachable | Files in the local store | Publisher retries with backoff; the disk budget applies and the oldest unshipped segments are never dropped before shipped ones |
| PUT fails | Object absent | Publisher retries that object; the plane cursor does not advance past a failed sequence |
| PUT response lost | Object may exist in S3 | Re-authorize; `stored` confirms the upload, `upload_required` means PUT again |
| Crash mid-flush | Cursor not advanced | Everything in the flush re-ships; every landed object returns `stored` |
| Disk pressure | Files in the local store | The profiler stops recording at its budget; the publisher frees shipped files first |
| Publisher token rotated | Files in the local store | 401; the publisher stops shipping, reloads config and retries with the new token |
| Local checksum failure on a file | File skipped and reported | The plane blocks until the file validates or a 60 s timeout passes; the projector then sees a gap and marks affected executions `index_corrupt` |
| Re-authorize finds a different checksum under a segment or pack key | Original object in S3 | Gateway returns `conflict`; publisher reports a local bug and never overwrites |
| Lost S3 notification | Object in S3, no message | The replay tool enqueues a synthetic message |
| Value referenced but not yet ingested | Meta message in the queue | The execution is `pending`; the message is retried until the pack lands |
| Stream stops without `RootEnded` | `RootStarted` shipped, no `RootEnded` | The sweep marks the execution `abandoned` after its limit |
| Malformed object with a valid checksum | Object plus quarantine marker | Message goes to the dead-letter queue; operators inspect and replay |
| Boundary or ClickHouse unavailable | Message in the queue | Retries with backoff after the visibility timeout; the queue is the buffer |
| Projector crashes mid-execution or loses a batch response | Some batches visible, no Lance commit | Redelivery repeats the same `batch_id` and `content_digest` and appends the same rows |
| Lance commit conflict | Fragments written, version not committed | Retry the commit against the newest version |
| Retry limit reached | Message in the dead-letter queue | Fix the cause, then replay the dead-letter queue |
| ClickHouse or Lance data lost | Objects in S3 | Replay every meta object and pack; ClickHouse is rebuilt into a new projection generation |
| Coordinator node or NVMe cache lost | Nothing lost | Warm replica serves queries; results are the same, latency is higher |

## Query execution

### Query sequence

```mermaid
sequenceDiagram
  participant U as Studio, CLI or agent
  participant Q as Query coordinator
  participant B as Boundary workload API
  participant CH as ClickHouse
  participant N as NVMe cache
  participant L as Lance datasets

  U->>Q: POST /v1/query with user token
  Note over Q,B: Coordinator initiates every HTTPS connection
  Q->>B: POST /query-snapshots
  B-->>Q: snapshot_id, catalog_version, generation, expires_at
  Q->>L: Read latest committed version
  Q->>Q: Plan with DataFusion, classify query
  Q->>B: POST /resident-queries
  B->>B: Reparse, inject scope, apply budget
  B->>CH: Execute resident subquery
  CH-->>B: Public rows
  B-->>Q: Arrow with generation in metadata
  Q->>N: Read private ranges through the cache
  N->>L: Fetch ranges missing from the cache
  Q-->>U: NDJSON rows then one queryOutcome
```

Studio requests originate in the user's browser. The coordinator must support TLS, Boundary user tokens and CORS for the Studio origin.

### Snapshot semantics

The coordinator binds one snapshot per query. The snapshot holds the tenant, project and environment, the catalog version, the active ClickHouse projection generation, the latest committed Lance dataset version and an expiry time for the Boundary part. The `snapshot_id`, `generation` and `expires_at` come from `POST /query-snapshots`. That call is cheap and the coordinator caches its result until `expires_at`. The Lance version is read from the bucket.

All resident subqueries and all Lance reads for one query use the same snapshot. A public fact whose private rows are not in the bound Lance version is reported with `private_state = pending`, never as `NULL` or a non-match.

### Resident subquery semantics

The coordinator sends only the public portion of the plan to `POST /resident-queries`. The request names either the `snapshot_id` or the `generation` with its `catalog_version`. Boundary reparses the SQL, permits registered relations and operations, injects authenticated tenant scope and applies query budgets. It does not run caller-provided ClickHouse SQL. The response is an Arrow IPC stream. It carries the generation in the `X-Baml-Projection-Generation` header and in the Arrow schema metadata. Private literals never appear in a resident subquery. A predicate such as `args['invoice']['id'] = 'INV-1042'` is evaluated in the coordinator.

### Private execution over Lance

Private operations run in the coordinator over the bound Lance version: typed column filters, n-gram text search, numeric ordering, grouping by private fields and reading a call's values. A call's value is read from the values dataset by CID, or from the CAS prefix by key when it is a large value. Public rows prune the private scan by function, time window, status and availability state. Lance reads go through a Foyer cache on local NVMe. The cache is disposable. Losing it changes latency, not results.

### Query classes and background jobs

The planner classifies each query. The numbers below are proof-of-concept measurements on one instance with 15 million synthetic calls. Cold is one controlled run with no disk cache. Warm is the worst of three runs from NVMe. They are targets, not SLAs.

| Class | Reads | Target | POC measurement |
|---|---|---|---|
| Public-only | ClickHouse only | Under 500 ms | 146 to 170 ms |
| Private, read a call's values | Public identity lookup, then Lance | Under 2 s cold, 1 s warm | 1,074 ms cold, 943 ms warm |
| Private or mixed search and grouping | Public pruning, then Lance | Under 2 to 5 s cold, 1 to 2 s warm | 1,331 to 1,606 ms cold, 959 to 1,055 ms warm |
| Background | Same plan, no interactive limit | Never interactive | Anti-join 9,705 ms cold, 2,293 ms warm |

Every query request carries a `budget` with wall time, result rows, result bytes and private bytes read. A query becomes background in two ways. The caller sends `mode: background` to `POST /v1/query-jobs`, or the planner's cost estimate exceeds the interactive budget. In the second case `POST /v1/query` returns an outcome with a typed `error` whose `remedy` points the caller to `POST /v1/query-jobs`.

A background job is accepted with status 202 and a job in state `queued`. It runs the same plan under the same snapshot rules. `GET /v1/query-jobs/{job_id}` reports `state`, `progress`, `rows_so_far`, the snapshot and `expires_at`. Once succeeded it also reports `result_url`. Once failed it reports `error`. `DELETE /v1/query-jobs/{job_id}` cancels a queued or running job. A job in a final state is returned unchanged.

The job writes its result as NDJSON with a final `queryOutcome` to the temporary prefix in the bucket. `GET /v1/query-jobs/{job_id}/result` redirects to or streams that file. It returns 409 while the job has not succeeded. The job record and its result are removed at `expires_at`.

### Streaming outcome

The public query response contains zero or more JSON row objects, one per NDJSON line, followed by exactly one `{"queryOutcome": ...}` record. Rows are not wrapped in a `row` property. The outcome carries:

| Field | Content |
|---|---|
| `queryCompleted` | Whether execution reached its natural end |
| `resultState` | `complete`, `incomplete`, `failed`, `budget_exhausted` or `cancelled` |
| `snapshot` | `catalogVersion`, `generation` and `lanceVersion`, plus `snapshotId` when known |
| `rowsStreamed` | Rows written before the outcome |
| `valueEvaluations` | Private values `attempted`, `available` and `unavailable`, with unavailable counts `byReason` |
| `privateState` | Rows counted as `indexed` and as `pending` |
| `error` | Typed error with `code`, `message`, `retryable` and an optional `remedy` |

An empty result is never produced by skipping unavailable evidence. If the stream ends before the outcome record, the result is incomplete. Once response bytes have been consumed, a retry starts a new public query and snapshot.

## Erasure

`POST /v1/erasures` accepts exactly one selector: `process_euids`, `execution_ids` or a `time_range`. It returns an erasure with `erasure_id` and `state`. `GET /v1/erasures/{erasure_id}` reports the state, `streams_matched` and `streams_deleted`. The erasure unit is the stream, identified by its process euid, because a data segment holds groups of several executions and cannot be split. A selected execution erases the whole stream that contains it. The gateway runs the erasure in this order:

1. Delete the stream's objects from S3, including prior versions: its meta segments, data segments and value packs.
2. Call `POST /projection-erasures` with `root_ids`, the root ids of every execution in the erased streams, and `generation`. Boundary deletes the facts idempotently.
3. Commit a new Lance version without the rows for those executions.

The stream is removed first, so a later replay cannot resurrect it. Erasure of `execution_ids` or a `time_range` resolves to process euids and root ids through the public facts before step 1. Large values and function tables are shared across streams and are removed only when no remaining stream in the scope references them. That needs a reference count over the remaining streams, which is an open implementation decision below.

## State machines

Object state per plane, computed from the local store, S3, the queue and the query stores:

```text
written locally -> shipped -> notified -> projected        meta segments and value packs
written locally -> shipped                                 data segments, large values, tables
```

Execution state:

```text
pending -> projected
       \-> abandoned
```

Projector progress for one queue message. These are steps, not stored states:

```text
received -> verified -> facts visible -> Lance version committed -> message deleted
                    \-> pending (message retried)
                    \-> quarantined (dead-letter queue)
```

Background job state:

```text
queued -> running -> succeeded
                 \-> failed
                 \-> cancelled
```

Erasure state:

```text
queued -> running -> succeeded
                 \-> failed
```

## Data formats

| Interface | Format | Reason |
|---|---|---|
| Control operations | HTTPS JSON | Small, inspectable metadata |
| Segment, large value and table upload | One S3 object per file, one zstd frame of the original bytes | Original bytes preserved; the profiler's reader runs unchanged over the bucket |
| Value pack | `BAMLPCK1` header, uncompressed canonical JSON manifest, then the raw value files; the whole pack is one zstd frame | Many small values in one PUT; one value readable alone by offset |
| Projection batch | zstd-compressed Arrow IPC stream | Typed, efficient columnar batches |
| Resident query response | Arrow IPC stream | Typed bulk transfer to the coordinator |
| Public query response | NDJSON stream | Incremental rows plus an explicit final outcome |
| Projection work | S3 event notification JSON on SQS, for the `.bamlmeta.zst` and `.bamlpack.zst` suffixes | At-least-once delivery with no custom message schema |
| Private data | Lance datasets under the derived prefix, including the values dataset keyed by CID | Versioned, rebuildable, indexable columnar storage |
| Background result | NDJSON file with a final outcome under the temporary prefix | Same reader as the interactive stream |

## Errors and retries

| Condition | Caller behavior |
|---|---|
| Invalid request or unsupported version | Do not retry without changing the request |
| Authentication or authorization failure | Refresh identity or correct scope before retrying; the publisher stops shipping and retries config later |
| Create-only PUT precondition failure | Re-authorize that object; accept `stored`; report `conflict` on a segment or pack, never overwrite |
| Object exceeds `max_object_bytes` | Report; the object is not shipped |
| Rate, budget or concurrency limit | Retry after server guidance with exponential backoff and jitter, or submit a job |
| Dependency unavailable | Retry idempotently with exponential backoff and jitter |
| Mutation timeout with unknown result | Retry with the same identifiers or read resource state |
| Query stream ends without an outcome | Treat the result as incomplete and start a new query |

Logs contain identifiers and workflow states. They do not contain presigned URLs, SQL value literals, evidence bodies or private values.

## Parallel implementation plan

### Control plane

- Implement workload authentication and scope validation.
- Define PlanetScale tables for tenants, projects, environments, deployments, tokens, catalog versions and the active generation.
- Implement Arrow projection ingestion with schema validation, deterministic batch IDs and digest conflicts.
- Implement `POST /projection-erasures` and `GET /projection-schemas/{catalog_version}`.
- Implement query snapshots.
- Implement resident SQL validation, scope injection, budgets and ClickHouse execution.

### Data plane

- Implement publisher authentication, `GET /v1/publisher/config`, and per-flush upload authorization at the gateway with one HEAD per object and create-only presigned URLs.
- Implement the publisher's collect, compress, authorize, PUT in order, cursor advance and retention loop, and the store operations it needs: list committed sequences per plane, mark shipped values and tables, release shipped segments and expire shipped values through the usage ledger, and report disk pressure.
- Configure S3 notifications with two rules on the projection queue, the queue itself and the dead-letter queue.
- Implement the SQS-triggered projector: pack ingest into the values dataset, `RootEnded` projection with the bounded walk-back, verification, deterministic Arrow batching, Lance append and commit, quarantine.
- Implement the index, compaction and sweep worker and the replay tool.
- Implement the query coordinator: DataFusion planning, plan splitting, Lance execution through the NVMe cache, NDJSON streaming, CORS and browser-facing authentication.
- Implement background jobs, result files and erasure orchestration.

### Shared integration artifacts

- OpenAPI-generated request and response types.
- Fixed JSON fixtures for every control operation, including an authorize request with all five object kinds.
- Fixed Arrow fixtures for every projected relation.
- Fixed object fixtures: a valid stream, a segment with a bad trailing checksum, a `RootEnded` whose pack has not landed, a stream without `RootEnded`.
- S3 notification message fixtures for both suffixes, including the replay tool's synthetic shape.
- Duplicate-delivery, projector-crash, lost-response, idempotency and conflict conformance tests.
- Tenant-isolation negative tests.
- A mock control-plane server for data-plane development and a mock data-plane client for control-plane development. Neither team depends on the other team's internal models.

## Open implementation decisions

OpenAPI examples do not resolve these decisions. Storage-format choices such as the mapping from files to objects, the pack format, object keys and Lance fragment size are recorded on the [Storage decisions](baml-studio-s3-design.html) page.

| Decision | Must be resolved before |
|---|---|
| Final permissions and endpoint-to-permission matrix | Authentication integration |
| Publisher-token format, issuance and rotation | Gateway integration |
| Shipped-value retention window and its disk-pressure override | Publisher integration |
| Walk-back bound for `RootStarted` and `EngineStarted` | Projector integration |
| Sweep interval and silence limit for closing executions as `abandoned` | Projector integration |
| Reference count over remaining streams for erasing shared large values and function tables | Erasure integration |
| Workload-token issuance and downscoping in shared Alien cells | Multi-tenant hosted integration |
| mTLS certificate provisioning and rotation | Control-plane deployment |
| Registered Arrow schemas and compatibility rules | Projection integration |
| Queue visibility timeout, maximum receives and dead-letter retention | Projector load testing |
| Lance index types, compaction cadence and version retention | Query performance testing |
| Interactive query budgets, job limits and result-file expiry | Query performance testing |
| Hosted S3 tenant isolation using buckets, prefixes or scoped roles | Hosted multi-tenant launch |
| Final error-code registry | Client SDK stabilization |
| Service limits and production SLOs | Production readiness review |

## Contract change process

1. Semantic changes start in this document.
2. Wire changes are made in the OpenAPI contract in the same change.
3. Both teams review changes that affect their caller or implementation responsibilities.
4. Generated types and shared fixtures are updated with the contract.
5. Contract tests must pass before either implementation depends on the new version.

Breaking changes require a new HTTP, catalog, projection schema or message version, depending on the affected surface.

The 2026-09-09 revision moves the unimplemented OpenAPI draft from `0.2.0-poc` to `0.3.0-proposal`. It removes the upload commit and projection dispatch endpoints (`/v1/uploads/{authorization_id}/commit`, `/evidence/commits`, `/evidence/commits/{commit_id}`, `/projection-dispatches/pending`, `/projection-dispatches/{commit_id}` and `/projections/commits/{commit_id}/complete`) and adds query jobs, erasures and projection schemas. The retired mechanisms are explained on the [Deferred decisions](deferred-signed-receipts.html) page. The 2026-09-10 revision keeps `0.3.0-proposal`: `POST /v1/uploads/authorize` now lists every object of a flush and answers per object, `GET /v1/publisher/config` is added, and the old draft's per-upload identifier, `profiler_schema_version`, `authorization_id` and `max_size_bytes` fields are gone; `time_range` survives only as an erasure selector. Draft-generated clients must regenerate types. This is not a migration of a deployed API.