BAML control-plane / data-plane contract

Status: Proposal, 2026-09-09. Revised 2026-09-10 for the publisher design. Supersedes the 2026-09-01 scale-out draft contract and the 2026-09-04 acceptance decision. It incorporates the query proof of concept (15 million synthetic calls, 13 queries) and the S3 storage experiment (2,328 execution roots, 12 upload comparisons, one accepted upload in the experiment's benchmark bundle format rebuilt into query data twice). The hosted system is not built.

Boundary services do not connect to the data plane. Data-plane workloads connect to Boundary over outbound HTTPS and send only registered non-value fields. Profiler and query clients connect to data-plane endpoints. Evidence, source material, arguments, outputs and errors stay in data-plane S3.

Scope

Customers keep evidence and exact values in their data plane. Boundary runs shared configuration and analytical services.

• The publisher calls the data-plane gateway. The publisher is the in-process background task inside the customer's application that ships the profiler's closed files to the cloud. The profiler is the in-process library that records calls and writes those files.
• Studio running in the browser, the CLI or an agent calls the data-plane query coordinator.
• Data-plane workloads initiate outbound HTTPS requests to Boundary.
• Boundary control-plane servers cannot initiate a connection into the data plane. Boundary replies on the connection the data plane opened.
• The data plane reads exact values and returns them only to the authorized query client.

This applies to customer-hosted BYOC and Boundary-hosted Alien data planes.

ALLOWED

Profiler publisher ----------------------> Data-plane gateway
Studio browser, CLI or agent -----------> Data-plane query coordinator
Data-plane workloads -- outbound HTTPS -> Boundary workload API
S3 object-created event ---------------> Data-plane projection queue (SQS)

NOT ALLOWED

Boundary control-plane server -----X----> Data plane

Data sent to Boundary

• Non-value configuration data: tenants, projects, environments, deployments, tokens, catalog versions and the active projection generation.
• Non-value analytical facts: execution and call identifiers, function names, revision IDs, status, timing, health counters and availability states such as args_state.¹ Token counts are not recorded today and would need a catalog version bump.
• Object keys and CIDs are opaque identifiers Boundary may hold; value bodies never. A CID is the content identifier of one value, a SHA-256 over the codec and the body.
• Resident subqueries over those facts, and root ids in erasure requests. A resident subquery is the public part of a query plan.

Data kept in the data plane

• Exact arguments, outputs and error values.
• Source text, source paths and snapshots.
• Meta segments, data segments, value packs, large values and function tables: the immutable objects the publisher writes to S3.
• Lance datasets with private typed columns, the values dataset (values_v1), and their scalar and n-gram indexes.
• Private literals and private field paths in query predicates, such as args['invoice']['id'] = 'INV-1042'.
• Background job result files.

Values stay in the data plane. Registered relation schemas define which non-value fields Boundary may process.

Contract scope

This document defines workflow behavior, state transitions, failure handling and ownership. The OpenAPI contract defines HTTP paths, methods, headers, bodies and status codes. Resolve conflicts before implementation.

The Storage decisions page records the recommended bucket layout, the mapping from profiler files to objects, the value pack format, the Lance layout and the queue settings, with alternatives. Its examples are design inputs. They do not add fields or endpoints to this contract. The publisher guide describes the publisher's algorithm and the profiler facts it relies on.

Identity and deployment model

tenant_id identifies a Boundary customer organization and is the root authorization boundary. A tenant owns projects, and a project owns environments.

PlanetScale is the configuration metastore. It is the system of record for this hierarchy, deployment assignments, tokens, catalog versions and the active projection generation. It holds no per-object and no per-flush state.

• A BYOC deployment is assigned to one tenant.
• A Boundary-hosted Alien cell may run deployments for multiple tenants. Sharing a cell does not change ownership.
• Every service validates scope against authenticated identity and PlanetScale.
• A scope supplied in a request body is not authority by itself.

Surfaces and ownership

The contract has two HTTP surfaces and one queue message.

The public data-plane API runs in a customer VPC or a Boundary-hosted Alien cell. Studio must reach it from the browser. A BYOC route may be public or use customer-network access such as a VPN or private-access proxy.

The control plane hosts the workload API. Data-plane workloads open every connection to it, and responses use the same connection.

The queue message is the standard S3 event notification JSON. The replay tool emits the same shape. The replay tool is an operator command that lists the meta objects and value packs under the evidence prefix, compares the root ids they carry with the projected root ids in ClickHouse and Lance, and enqueues a message for each object whose executions or values are missing.

Authentication

The contract defines three authentication contexts:

1. A Boundary user token authorizes Studio, the CLI or an agent to call the query coordinator for one tenant, project and environment. Erasure requires an additional erasure permission on the token.
2. A publisher token authorizes the publisher to call GET /v1/publisher/config and POST /v1/uploads/authorize at the gateway for one environment. It is issued by the control API and rotated by the customer. A 401 or 403 response means the token is invalid: the publisher stops shipping, keeps its files within the disk budget, and retries the config endpoint later.
3. A data-plane workload identity uses mTLS and a short-lived workload JWT to call the Boundary workload API as a deployment and cell.

Every request carries a caller-generated X-Baml-Request-Id header. Mutations that create a job, an erasure or a projection erasure also carry an Idempotency-Key header. Repeating a request with the same key returns the same resource.

OpenAPI permission names are provisional and non-exhaustive. Define the final permission set and endpoint matrix during implementation.

Contract invariants

• Boundary never initiates a connection into the data plane.
• Only non-values cross into Boundary. Values remain in data-plane S3 and in derived Lance datasets in the same bucket.
• Scope is derived from authenticated identity and PlanetScale, not request JSON alone.
• An object in S3 is immutable. Its key is a pure function of scope, stream, plane and sequence, or of scope and CID, and its checksum identifies it forever. A successful S3 PUT is the only upload success signal.
• A meta object's notification fires only after every data segment it references and every pack, large value and table those segments reference exist in the bucket, because the publisher uploads in that order. There are no per-upload records in Boundary.
• Notification and queue delivery are at least once. Projection is deterministic and duplicate-safe. An execution is projected exactly once per generation, keyed by its root id.
• A query binds one generation, one catalog version and one Lance version for its whole execution. Unavailable evidence is reported in queryOutcome, never silently dropped. A streamed result is incomplete without its final outcome record.
• ClickHouse facts, Lance datasets, indexes, the NVMe cache and queue messages are rebuildable from the bucket.

Endpoint directory

Evidence ingestion

Storage model

Three systems answer three different questions:

S3 is the canonical evidence store and mirrors the profiler's local stream layout. The queue carries work, not state. ClickHouse and Lance are derived and rebuildable. No system outside the bucket records that an individual upload happened.

Object kinds

A stream is one process's file sequence, identified by a 32-hex process euid. A segment is one immutable file in a stream plane, meta or data. The publisher ships five kinds of object. The kind value is the one used in the authorize request.

<seq> is 20 decimal digits. <cid> is the 64-hex digest without the bamlv_1_ prefix. A value pack holds the small values of one flush. A large value is a value at or above the large-value threshold and is stored as its own object. A function table is a CAS object with codec 2. Every object, packs included, is one zstd frame of its original bytes.

Upload sequence

The step numbers match the ingest flow on the architecture page. A flush is one publisher cycle.

sequenceDiagram
  participant U as Publisher
  participant G as Gateway
  participant S3 as S3
  participant Q as SQS
  participant P as Projector
  participant B as Boundary ingest API
  participant CH as ClickHouse
  participant L as Lance

  U->>U: 1-2. Profiler closes files; publisher collects the flush
  U->>G: 3. POST /v1/uploads/authorize with every object
  G->>S3: 4. HEAD each key
  G-->>U: 4. Per object: create-only presigned PUT URL, or stored
  U->>S3: 5. PUT tables, large values, packs, then data, then meta
  S3-->>U: 5. Stored durably
  U->>U: 6. Advance cursor, mark values, release segments
  S3->>Q: 7. Object-created for each meta segment and pack
  Q->>P: 8. Deliver pack message
  P->>L: 8. Ingest entries into the values dataset by CID
  Q->>P: 9. Deliver meta message
  P->>S3: 9. Read data range, table and values by key
  P->>B: 10. PUT /projection-batches/{batch_id}
  B->>CH: 10. Insert facts
  B-->>P: 10. Batch visible
  P->>L: 11. Commit new dataset version
  P->>Q: 11. Delete message
  Note over P,L: 12. Sweep closes abandoned executions; index and compaction run

The publisher runs a flush every flush_interval_seconds, earlier when pending pack bytes exceed pack_max_entry_bytes, and on shutdown. Nothing waits for Boundary. The flush is complete when the meta segments are stored.

Publisher configuration

GET /v1/publisher/config is called at start and every five minutes, and the response is cacheable. It returns capture_enabled, supported_segment_versions ([2]), supported_cas_versions ([1]), flush_interval_seconds (5), max_objects_per_authorize (100), large_value_threshold_bytes (32 KiB), max_object_bytes (256 MiB), pack_max_entry_bytes (8 MiB), shipped_value_retention_seconds (3,600) and upload_url_ttl_seconds (900). The values in parentheses are starting values. Changing one changes publisher behavior without a runtime release. The OpenAPI contract describes every field.

Authorize request and response

One POST /v1/uploads/authorize per flush lists every object the flush wants to write, at most max_objects_per_authorize per call. The request carries process_euid, flush_id and objects. Every entry carries kind, size_bytes and checksum_sha256 of the zstd object. Segments add sequence, packs add entry_count, and values and tables add cid and codec.

The gateway validates the publisher token, computes each key from the authenticated scope and the entry, and checks S3 for each key. The response repeats flush_id and objects in the same order. Every entry carries kind, its sequence or cid, its key and a state:

required_headers carries If-None-Match: *, x-amz-checksum-sha256 and Content-Length, so the URL is bound to that key, size and checksum. A 401 or 403 stops shipping. A 413 means one object exceeds max_object_bytes or the request lists more than max_objects_per_authorize objects; the publisher splits the request or drops the oversize object. A 429 or 503 means back off with jitter.

Object and execution states

Each object passes through four states per plane. None of them is recorded in Boundary.

An execution has three states, computed from the stream and the query stores. It is pending once its RootEnded has been seen but the execution is not yet projected, including while a value it references is not yet ingested. Before RootEnded arrives the execution is not known to the cloud at all; the catalog reports it as running only for local reads. It is projected when its public fact batches are visible in ClickHouse and its private rows are committed in Lance. It is abandoned when its stream stopped without RootEnded and the sweep closed it.

The coordinator computes private_state from the bound Lance version. A public fact whose private rows are not in the bound Lance version carries private_state = pending; once they are, it carries private_state = indexed. Background index and compaction work does not change object or execution state.

Idempotency

Upload retries:

• Keys are pure functions of scope, stream, plane and sequence, or of scope and CID. The publisher does not choose keys and cannot collide with itself.
• A repeat flush re-authorizes with the same objects and gets stored for every object that landed. A crash before the cursor advances re-ships the whole flush; duplicates are harmless. An expired URL is replaced the same way, and expiry never permits a different key or different bytes.
• The presigned URL is create-only (If-None-Match: *). A PUT that fails the create-only precondition with 412 means the object already exists. The publisher re-authorizes that object and expects stored.
• conflict on a segment or a pack is a local bug. The publisher reports it and never overwrites. conflict on a value or a table is treated as stored.
• A pack may hold a value that an earlier pack or another stream already shipped. The projector collapses duplicates by CID when it ingests the pack.

Projection batches:

• batch_id is deterministic from the root id, relation and partition. Fact row IDs are deterministic from the same inputs plus the call position. The generation travels in a header, so a rebuild into a new generation reuses the same batch IDs.
• A PUT /projection-batches/{batch_id} whose content_digest matches a batch already visible in that generation returns the existing visible state. A reused batch_id with a different content_digest is rejected as a conflict.
• Lance row IDs are deterministic from the root id, relation and partition. The values dataset is keyed by CID. The projector merges on the row ID, so a duplicate run produces one row set, not two.
• A duplicate notification or a projector crash mid-execution is completed by redelivery with the same IDs and the same result.

Projection contract

The projector receives an S3 notification, not an evidence body. The message names the bucket, object.key, object.size, object.eTag and object.versionId. The key suffix selects the handling.

On a value pack (.bamlpack.zst): the projector decompresses the pack, reads the manifest and ingests each entry into the values dataset in Lance, keyed by CID with codec, body and locator. Duplicates collapse. No public fact leaves the data plane.

On a meta segment (.bamlmeta.zst): the projector decodes the segment. For each RootEnded record it:

1. reads meta sequence 1 of the stream by key, because StreamStarted is record 1 of meta sequence 1 and every timestamp in the stream is StreamStarted.zero_unix_ns plus a nanosecond offset; then walks back through earlier meta objects of the same stream, by key in descending sequence, to find the execution's RootStarted and the EngineStarted for its engine. Only those two records need the walk. The walk is bounded to 2,000 meta objects. A walk that exhausts the bound without finding both records is treated like a sequence gap and marks the execution index_corrupt;
2. reads data segments [data_first_seq ..= data_last_seq] of the stream by key, the range carried by RootEnded, and skips groups that belong to other roots;
3. reads the function table by the CID in EngineStarted.function_table_cid;
4. resolves each value CID from the values dataset, else from the cas/values/ prefix by key, else reports the execution pending and leaves the message for a retry;
5. verifies that every key lies under the evidence prefix of its own deployment scope, that every file's trailing checksum matches, that the segment versions are supported, and that the active projection generation from Boundary is the one it will write to. An S3 checksum alone does not qualify an object; the storage experiment showed that it cannot prove complete dependencies, which here are the data range, the table and the values;
6. emits ClickHouse batches and Lance rows with deterministic IDs keyed by the root id, and deletes the message once every batch is visible and the Lance version is committed.

The projector produces two outputs per execution:

The availability states are written as facts with the catalog's values. In calls_v1, args_state, output_state and error_state take exactly available, not_captured, not_applicable or lost:<reason>, with reasons such as value_too_large and queue_full; a value is whole or lost with a reason, never truncated. In threads_v1, status takes succeeded, failed, cancelled, panicked, abandoned, or running for a live stream; value_state takes none, partial or complete; and index_state is the meta-plane integrity state, complete, index_corrupt, root_started_lost or no_root_ended. program_id is random per engine today, so call_path_id values are not comparable across processes and function_definitions_v1 grows by one row set per engine start until the planned content-hash program id lands.

Each batch request carries its identity in headers:

Boundary accepts only the registered relation schema. The projector reads the schema for its catalog version from GET /projection-schemas/{catalog_version}. Value-bearing fields, source material, unknown fields and unapproved string columns are rejected.

The Lance commit is the last step. It appends the new fragments and commits a new dataset version. Two projectors committing at once produce a conflict. The loser retries the commit against the newest version. Fragments are never rewritten on retry.

Executions whose stream stops without RootEnded are closed by the sweep. The sweep is the scheduled job that lists the stream prefixes under the evidence prefix, finds the streams whose newest meta object is older than the silence limit, and marks their executions with a RootStarted and no matching RootEnded as abandoned with the facts it has. The sweep interval and the silence limit are open implementation decisions below. An interior sequence gap in a plane is corruption: the projector marks every execution whose data range crosses the gap index_corrupt.

An object that fails verification is quarantined. The projector sends the message straight to the dead-letter queue and writes a quarantine marker under quarantine/<original key>.quarantine. The object bytes are kept. Operators inspect and replay after a fix.

Failure handling

S3, SQS, ClickHouse and Lance do not share a transaction. Determinism and redelivery repair every incomplete transition.

Query execution

Query sequence

sequenceDiagram
  participant U as Studio, CLI or agent
  participant Q as Query coordinator
  participant B as Boundary workload API
  participant CH as ClickHouse
  participant N as NVMe cache
  participant L as Lance datasets

  U->>Q: POST /v1/query with user token
  Note over Q,B: Coordinator initiates every HTTPS connection
  Q->>B: POST /query-snapshots
  B-->>Q: snapshot_id, catalog_version, generation, expires_at
  Q->>L: Read latest committed version
  Q->>Q: Plan with DataFusion, classify query
  Q->>B: POST /resident-queries
  B->>B: Reparse, inject scope, apply budget
  B->>CH: Execute resident subquery
  CH-->>B: Public rows
  B-->>Q: Arrow with generation in metadata
  Q->>N: Read private ranges through the cache
  N->>L: Fetch ranges missing from the cache
  Q-->>U: NDJSON rows then one queryOutcome

Studio requests originate in the user's browser. The coordinator must support TLS, Boundary user tokens and CORS for the Studio origin.

Snapshot semantics

The coordinator binds one snapshot per query. The snapshot holds the tenant, project and environment, the catalog version, the active ClickHouse projection generation, the latest committed Lance dataset version and an expiry time for the Boundary part. The snapshot_id, generation and expires_at come from POST /query-snapshots. That call is cheap and the coordinator caches its result until expires_at. The Lance version is read from the bucket.

All resident subqueries and all Lance reads for one query use the same snapshot. A public fact whose private rows are not in the bound Lance version is reported with private_state = pending, never as NULL or a non-match.

Resident subquery semantics

The coordinator sends only the public portion of the plan to POST /resident-queries. The request names either the snapshot_id or the generation with its catalog_version. Boundary reparses the SQL, permits registered relations and operations, injects authenticated tenant scope and applies query budgets. It does not run caller-provided ClickHouse SQL. The response is an Arrow IPC stream. It carries the generation in the X-Baml-Projection-Generation header and in the Arrow schema metadata. Private literals never appear in a resident subquery. A predicate such as args['invoice']['id'] = 'INV-1042' is evaluated in the coordinator.

Private execution over Lance

Private operations run in the coordinator over the bound Lance version: typed column filters, n-gram text search, numeric ordering, grouping by private fields and reading a call's values. A call's value is read from the values dataset by CID, or from the CAS prefix by key when it is a large value. Public rows prune the private scan by function, time window, status and availability state. Lance reads go through a Foyer cache on local NVMe. The cache is disposable. Losing it changes latency, not results.

Query classes and background jobs

The planner classifies each query. The numbers below are proof-of-concept measurements on one instance with 15 million synthetic calls. Cold is one controlled run with no disk cache. Warm is the worst of three runs from NVMe. They are targets, not SLAs.

Every query request carries a budget with wall time, result rows, result bytes and private bytes read. A query becomes background in two ways. The caller sends mode: background to POST /v1/query-jobs, or the planner's cost estimate exceeds the interactive budget. In the second case POST /v1/query returns an outcome with a typed error whose remedy points the caller to POST /v1/query-jobs.

A background job is accepted with status 202 and a job in state queued. It runs the same plan under the same snapshot rules. GET /v1/query-jobs/{job_id} reports state, progress, rows_so_far, the snapshot and expires_at. Once succeeded it also reports result_url. Once failed it reports error. DELETE /v1/query-jobs/{job_id} cancels a queued or running job. A job in a final state is returned unchanged.

The job writes its result as NDJSON with a final queryOutcome to the temporary prefix in the bucket. GET /v1/query-jobs/{job_id}/result redirects to or streams that file. It returns 409 while the job has not succeeded. The job record and its result are removed at expires_at.

Streaming outcome

The public query response contains zero or more JSON row objects, one per NDJSON line, followed by exactly one {"queryOutcome": ...} record. Rows are not wrapped in a row property. The outcome carries:

An empty result is never produced by skipping unavailable evidence. If the stream ends before the outcome record, the result is incomplete. Once response bytes have been consumed, a retry starts a new public query and snapshot.

Erasure

POST /v1/erasures accepts exactly one selector: process_euids, execution_ids or a time_range. It returns an erasure with erasure_id and state. GET /v1/erasures/{erasure_id} reports the state, streams_matched and streams_deleted. The erasure unit is the stream, identified by its process euid, because a data segment holds groups of several executions and cannot be split. A selected execution erases the whole stream that contains it. The gateway runs the erasure in this order:

1. Delete the stream's objects from S3, including prior versions: its meta segments, data segments and value packs.
2. Call POST /projection-erasures with root_ids, the root ids of every execution in the erased streams, and generation. Boundary deletes the facts idempotently.
3. Commit a new Lance version without the rows for those executions.

The stream is removed first, so a later replay cannot resurrect it. Erasure of execution_ids or a time_range resolves to process euids and root ids through the public facts before step 1. Large values and function tables are shared across streams and are removed only when no remaining stream in the scope references them. That needs a reference count over the remaining streams, which is an open implementation decision below.

State machines

Object state per plane, computed from the local store, S3, the queue and the query stores:

written locally -> shipped -> notified -> projected        meta segments and value packs
written locally -> shipped                                 data segments, large values, tables

Execution state:

pending -> projected
       \-> abandoned

Projector progress for one queue message. These are steps, not stored states:

received -> verified -> facts visible -> Lance version committed -> message deleted
                    \-> pending (message retried)
                    \-> quarantined (dead-letter queue)

Background job state:

queued -> running -> succeeded
                 \-> failed
                 \-> cancelled

Erasure state:

queued -> running -> succeeded
                 \-> failed

Data formats

Errors and retries

Logs contain identifiers and workflow states. They do not contain presigned URLs, SQL value literals, evidence bodies or private values.

Parallel implementation plan

Control plane

• Implement workload authentication and scope validation.
• Define PlanetScale tables for tenants, projects, environments, deployments, tokens, catalog versions and the active generation.
• Implement Arrow projection ingestion with schema validation, deterministic batch IDs and digest conflicts.
• Implement POST /projection-erasures and GET /projection-schemas/{catalog_version}.
• Implement query snapshots.
• Implement resident SQL validation, scope injection, budgets and ClickHouse execution.

Data plane

• Implement publisher authentication, GET /v1/publisher/config, and per-flush upload authorization at the gateway with one HEAD per object and create-only presigned URLs.
• Implement the publisher's collect, compress, authorize, PUT in order, cursor advance and retention loop, and the store operations it needs: list committed sequences per plane, mark shipped values and tables, release shipped segments and expire shipped values through the usage ledger, and report disk pressure.
• Configure S3 notifications with two rules on the projection queue, the queue itself and the dead-letter queue.
• Implement the SQS-triggered projector: pack ingest into the values dataset, RootEnded projection with the bounded walk-back, verification, deterministic Arrow batching, Lance append and commit, quarantine.
• Implement the index, compaction and sweep worker and the replay tool.
• Implement the query coordinator: DataFusion planning, plan splitting, Lance execution through the NVMe cache, NDJSON streaming, CORS and browser-facing authentication.
• Implement background jobs, result files and erasure orchestration.

Shared integration artifacts

• OpenAPI-generated request and response types.
• Fixed JSON fixtures for every control operation, including an authorize request with all five object kinds.
• Fixed Arrow fixtures for every projected relation.
• Fixed object fixtures: a valid stream, a segment with a bad trailing checksum, a RootEnded whose pack has not landed, a stream without RootEnded.
• S3 notification message fixtures for both suffixes, including the replay tool's synthetic shape.
• Duplicate-delivery, projector-crash, lost-response, idempotency and conflict conformance tests.
• Tenant-isolation negative tests.
• A mock control-plane server for data-plane development and a mock data-plane client for control-plane development. Neither team depends on the other team's internal models.

Open implementation decisions

OpenAPI examples do not resolve these decisions. Storage-format choices such as the mapping from files to objects, the pack format, object keys and Lance fragment size are recorded on the Storage decisions page.

Contract change process

1. Semantic changes start in this document.
2. Wire changes are made in the OpenAPI contract in the same change.
3. Both teams review changes that affect their caller or implementation responsibilities.
4. Generated types and shared fixtures are updated with the contract.
5. Contract tests must pass before either implementation depends on the new version.

Breaking changes require a new HTTP, catalog, projection schema or message version, depending on the affected surface.

The 2026-09-09 revision moves the unimplemented OpenAPI draft from 0.2.0-poc to 0.3.0-proposal. It removes the upload commit and projection dispatch endpoints (/v1/uploads/{authorization_id}/commit, /evidence/commits, /evidence/commits/{commit_id}, /projection-dispatches/pending, /projection-dispatches/{commit_id} and /projections/commits/{commit_id}/complete) and adds query jobs, erasures and projection schemas. The retired mechanisms are explained on the Deferred decisions page. The 2026-09-10 revision keeps 0.3.0-proposal: POST /v1/uploads/authorize now lists every object of a flush and answers per object, GET /v1/publisher/config is added, and the old draft's per-upload identifier, profiler_schema_version, authorization_id and max_size_bytes fields are gone; time_range survives only as an erasure selector. Draft-generated clients must regenerate types. This is not a migration of a deployed API.