BAML Studio hosted architecture

Status: Proposal, 2026-09-09. Revised 2026-09-10 for the publisher design. Supersedes the 2026-09-01 scale-out proposal and the 2026-09-04 acceptance decision. It incorporates the query proof of concept (15 million synthetic calls, 13 queries) and the S3 storage experiment (2,328 execution roots, 12 upload comparisons, one accepted upload in the experiment's bundle format rebuilt into query data twice). The hosted system is not built.

The profiler records every BAML call inside the customer's application and writes closed files into a local store. The publisher is the in-process background task that ships the profiler's closed files to the customer's S3 bucket. A meta segment is one immutable file in a stream's meta plane; it carries the records that start and end executions. A data segment is one immutable file in a stream's data plane; it carries the call tree deltas, spans, value references and error captures for the executions in flight. A value pack is one object holding many small values from one flush. A large value is a value at or above the large-value threshold, stored as its own content-addressed object. A function table is a content-addressed object that maps function identifiers to their definitions. S3 tells a queue when a meta segment or a value pack arrives. A projector reads the objects and checks them. It sends allowlisted public facts to Boundary's ClickHouse. It appends private values and their indexes to Lance datasets in the customer's bucket. Lance is a columnar storage format with versioned datasets and built-in indexes. Studio, the CLI and agents run plain SQL against one logical catalog. A query coordinator in the customer's cloud plans each query. It runs the public part in ClickHouse and the private part over Lance. It streams rows with one typed outcome.

Nothing in Boundary records individual uploads. S3 is the evidence, the queue carries the work, and everything else can be rebuilt from the bucket.

The public query language is portable read-only SQL with no BAML-specific dialect. DataFusion, an embeddable SQL engine, plans every query. baml query executes the same SQL from the CLI. The catalog hides the ClickHouse and Lance split. This proposal treats function names and revision IDs as public facts.¹

Decisions

1. Sensitive evidence stays in the customer's cloud. Customer S3 is the canonical evidence store.
2. Boundary operates the multi-tenant control plane: control API, configuration metastore on PlanetScale, projection ingest API, resident query API and ClickHouse Cloud.
3. ClickHouse stores only allowlisted, non-value public facts: execution and call identifiers, function names, revision IDs, status, timing and availability states such as args_state. The calls_v1 columns args_state, output_state and error_state take exactly available, not_captured, not_applicable or lost:<reason>, with reasons such as value_too_large and queue_full. A value is whole or lost with a reason; the profiler never truncates. Token counts are not recorded today and would need a catalog version bump.
4. Private values live only in the customer's bucket. Canonically they live in value packs and large-value objects under the evidence prefix. Derived copies live in rebuildable Lance datasets with private typed columns and n-gram indexes, including the values dataset (values_v1) keyed by CID.
5. There are no per-upload records in Boundary. A successful S3 PUT is the end of the upload.
6. S3 mirrors the profiler's local stream layout; the projector runs the existing stream reader over an object-store backend. Every object key is a pure function of scope, stream, plane and sequence, or of scope and CID.
7. Work delivery is S3 event notifications into SQS Standard with a dead-letter queue. Only meta segments and value packs notify. Delivery is at least once. Projection is deterministic and duplicate-safe.
8. The public query language is portable read-only SQL planned by DataFusion over one versioned logical catalog. Catalog v1 exposes six public relations, threads_v1, call_path_stats_v1, calls_v1, errors_v1, function_definitions_v1 and health_v1, plus the internal processes_v1, store_files_v1 and value_index_v1.
9. Queries are classified as interactive (public-only, private, mixed) or background. Background jobs expose progress, cancellation and a downloadable result.
10. Everything outside the bucket is disposable. ClickHouse facts, Lance datasets, indexes, the NVMe cache and queue messages can all be rebuilt or replayed from the segments, packs and CAS objects.

Topology

flowchart LR
  subgraph Runtime[Customer application]
    PROF["Profiler<br/>In-process native library"]
    UP["Publisher<br/>In-process background task"]
  end

  subgraph DataPlane[Customer cloud]
    GW["Gateway<br/>Alien Worker · AWS Lambda"]
    S3[("Evidence bucket<br/>segments, packs, CAS")]
    LANCE[("Lance datasets and indexes<br/>Same bucket, derived prefix")]
    Q[("Projection queue<br/>Alien Queue · SQS Standard + DLQ")]
    PROJ["Projector<br/>Alien Worker · SQS-triggered Lambda"]
    IDX["Index, compaction and sweep worker<br/>Scheduled container"]
    QUERY["Query coordinator<br/>Alien Container · EC2 with NVMe cache"]
  end

  subgraph ControlPlane[Boundary cloud]
    CTRL["Control API<br/>Boundary service"]
    META[("Configuration metastore<br/>PlanetScale")]
    INGEST["Projection ingest API<br/>Boundary service"]
    CQUERY["Resident query API<br/>Boundary service"]
    CH[("Public facts<br/>ClickHouse Cloud")]
  end

  UI["Studio, CLI or agent<br/>Client application"]

  PROF --> UP
  UP -->|authorize flush| GW
  UP -->|presigned create-only PUTs| S3
  S3 -->|meta and pack notifications| Q
  Q --> PROJ
  PROJ -->|read segments, packs, CAS by key| S3
  PROJ -->|public fact batches| INGEST
  INGEST --> CH
  PROJ -->|private rows and index data| LANCE
  IDX -->|index, compact, prune versions, sweep| LANCE
  UI -->|Boundary user token| QUERY
  QUERY -->|query snapshot| CTRL
  QUERY -->|resident subquery| CQUERY
  CQUERY --> CH
  QUERY -->|Lance reads through NVMe cache| LANCE
  CTRL --> META

Alien is the data-plane runtime. Alien Workers run on AWS Lambda. The query coordinator runs as an Alien Container on an EC2 instance with local NVMe, for example the m7gd or i4g class. At least one warm replica runs beside it. Customer services connect to Boundary over outbound HTTPS only. Boundary never connects into the customer cloud. Studio in the browser calls the query coordinator directly over TLS with a Boundary user token.

Data placement

The storage decisions page records the recommended bucket layout, object layout, value pack format, Lance layout and queue settings, with alternatives. The placement below holds for every option on that page.

ClickHouse stores function names and revision IDs directly. Queries can filter, group and return them without a customer-local lookup. Source code, paths and exact values stay in the customer's bucket.

Three version numbers appear on this page.

ClickHouse must not contain:

• argument, output, error or log bodies;
• decoded BAML values or JSON;
• plaintext previews;
• source code or source paths;
• value-derived text, token, n-gram or vector indexes; or
• query literals that apply to private values.

Components

Customer application

Customer data plane

Boundary control plane

A resident subquery is the public part of a query plan. The coordinator sends it to Boundary, and Boundary runs it on ClickHouse. Foyer is the cache library the coordinator uses for Lance reads. It keeps hot ranges in memory and on local NVMe. No customer service receives direct metastore or ClickHouse credentials. The endpoint shapes are in the system contract page.

Ingest flow

1. The profiler writes closed segment files and value files into the local store. A file is either complete or absent.
2. Every flush interval, the publisher collects every complete file not yet shipped: meta segments, data segments, small values into one value pack, and large values and function tables as single objects.
3. The publisher compresses each object with zstd and calls the gateway once per flush with the list of objects, their sizes and their checksums.
4. The gateway validates the publisher token, computes each object key from the authenticated scope, checks S3 for each key, and answers per object: a create-only presigned PUT URL when the object is missing, or stored when it is already there with the same checksum.
5. The publisher PUTs the function tables, large values and value packs first, then the data segments in sequence order, then the meta segments.
6. After the meta segments are stored, the publisher advances its cursor, marks the shipped values, and releases the shipped segments to the local store. The flush is complete. Nothing waits for Boundary.
7. S3 sends an object-created notification to the projection queue for each meta segment and each value pack. Data segments and CAS objects do not notify.
8. A projector that receives a value pack notification ingests its entries into the values dataset in Lance, keyed by CID, so duplicates collapse.
9. A projector that receives a meta segment notification decodes it. For each RootEnded record it reads that execution's data segments by key, its function table by CID, and its values from the values dataset or from the CAS prefix.
10. The projector builds deterministic public fact batches and sends them to the Boundary projection ingest API, which writes them to ClickHouse and answers only when they are visible.
11. The projector commits the execution's private rows as a new Lance dataset version and deletes the queue message. A failure at any step leaves the message to be redelivered; after the retry limit it moves to the dead-letter queue.
12. In the background, the sweep closes executions whose stream stopped without RootEnded as abandoned, and the index and compaction worker updates indexes and merges small fragments.

The upload order in step 5 is the ordering guarantee. A RootEnded record carries the sequence range of the data segments that hold its execution, and those data segments reference values and a function table by CID. Because tables, large values and packs land first, data segments next and meta segments last, a meta object never appears in the bucket before the objects its executions need. Only meta segments and value packs notify because those are the two objects with work behind them: a meta segment may carry RootEnded, and a pack carries values to ingest. Data segments and CAS objects are read on demand by key, so a notification for them would carry no work.

An execution's RootStarted and its engine's EngineStarted are in the same meta segment as its RootEnded or in an earlier one of the same stream. The projector walks back through earlier meta objects by key, in descending sequence, bounded to 2,000 meta objects. Every timestamp in a stream is StreamStarted.zero_unix_ns plus a nanosecond offset, and StreamStarted is record 1 of meta sequence 1, so the projector reads meta sequence 1 by key; only RootStarted and EngineStarted need the bounded backward walk. A walk that exhausts the bound without finding both records is treated like a sequence gap.

Object layout

evidence/v1/<tenant>/<project>/<environment>/
  streams/<process_euid>/meta/<seq>.bamlmeta.zst      notifies the projection queue
  streams/<process_euid>/data/<seq>.bamldata.zst      read by key
  streams/<process_euid>/values/<flush_ulid>.bamlpack.zst   notifies the projection queue
  cas/values/<2 hex>/<cid>.bamlvalue.zst              large values, gateway-checked, read by key
  cas/tables/<2 hex>/<cid>.bamltable.zst              function tables, gateway-checked, read by key
  quarantine/<original key>.quarantine                marker written by the projector
derived/v1/<scope>/<dataset>/                         Lance datasets, internal to the data plane
temporary/v1/<scope>/jobs/<job_id>/result.ndjson      background job results

<seq> is 20 decimal digits. <cid> is the 64-hex digest without the bamlv_1_ prefix. No date partition; retention uses object age. Every object is one zstd frame. A segment, large value or table object is the original file compressed. A value pack is a small header, an uncompressed canonical JSON manifest and the raw value files, compressed together. S3 verifies the SHA-256 of the compressed object on PUT. The CID identifies the decompressed body of a value or table.

What each object family feeds (the public and private split is this proposal's; the profiler has no classification):

program_id is random per engine today, so call_path_id values are not comparable across processes and function_definitions_v1 grows by one row set per engine start until the planned content-hash program id lands.

Object and execution states

An object passes through up to four states. None of them is recorded in Boundary. The only publisher state is the local filesystem: a cursor per plane, and a marker next to each shipped value and table.

An execution has its own state, computed by the projector. It is pending when its RootEnded has been seen but the execution is not yet projected, for example because a referenced value has not been ingested; the message stays in the queue. It is projected when its public facts are visible in ClickHouse and the Lance version with its private rows is committed. It is abandoned when its stream stopped without RootEnded and the sweep closed it after the limit. In threads_v1, status takes succeeded, failed, cancelled, panicked, abandoned, or running for a live stream; value_state takes none, partial or complete; and index_state is the meta-plane integrity state, complete, index_corrupt, root_started_lost or no_root_ended. private_state is a per-row value the coordinator computes from the Lance version a query binds; rows of a projected execution report private_state = indexed once the coordinator binds that version, and private_state = pending before.

What replaced the commit ledger

The earlier proposals recorded every upload in Boundary and drove projection from that record. This proposal removes that record. S3 durability plus a checksum-verified, create-only PUT already gives the publisher a durable success signal. SQS with a dead-letter queue already gives at-least-once work delivery with a visible failure lane. The experiments also showed that upload metadata could not prove an upload was valid: in the experiment's bundle format, a valid checksum still arrived with a malformed manifest. Validation therefore belongs in the projector, which reads the bytes. The retired mechanisms and the conditions for reconsidering them are on the deferred decisions page.

Query flow

1. Studio, the CLI or an agent obtains a short-lived Boundary user token scoped to tenant, project and environment.
2. The client sends SQL to the query coordinator in the customer cloud.
3. The coordinator validates the token and binds a snapshot: the active ClickHouse projection generation and catalog version from Boundary, and the latest committed Lance dataset version from the bucket.
4. DataFusion parses and plans the SQL against the logical catalog and classifies the query: public-only, private, mixed, or background (by estimated cost or by caller request).
5. Public operations go to the Boundary resident query API as a resident subquery. Boundary reparses it, injects scope and budgets, runs it on ClickHouse, returns Arrow rows. Public-only queries end here.
6. Private operations run in the coordinator over Lance: typed column filters, n-gram text search, numeric ordering, grouping by private fields, and reading a call's values. Public rows prune the private scan (function, time window, status, availability).
7. The coordinator joins, orders and limits, then streams NDJSON rows followed by exactly one queryOutcome record with the snapshot, generation, Lance version, rows streamed, values evaluated, unavailable-value reasons and any typed error.
8. A background query returns a job ID immediately. The job runs the same plan, reports progress, can be cancelled, and writes its result to the temporary prefix in the bucket for download.

Job states are queued, running, succeeded, failed and cancelled. The endpoints are POST /v1/query for interactive queries, POST /v1/query-jobs to start a job, GET /v1/query-jobs/{job_id} for state and progress, DELETE /v1/query-jobs/{job_id} to cancel, and GET /v1/query-jobs/{job_id}/result for the stored result.

sequenceDiagram
  participant C as Studio, CLI or agent
  participant Q as Query coordinator
  participant B as Boundary
  participant L as Lance in customer S3
  C->>Q: POST /v1/query with SQL and user token
  Q->>B: query snapshot: generation and catalog version
  Q->>L: latest Lance dataset version
  Note over Q: DataFusion plans and classifies the query
  Q->>B: resident subquery for public operations
  B-->>Q: Arrow rows from ClickHouse
  Q->>L: private filters, search and ordering through the NVMe cache
  L-->>Q: private rows
  Q-->>C: NDJSON rows, then one queryOutcome

Public-only queries never read Lance. Private and mixed queries read Lance only within the set that public facts have pruned. The coordinator evaluates private operations before the final ORDER BY and LIMIT when pushing the limit down could change the result. Interactive queries carry a budget. A query whose estimated cost exceeds it is rejected with a typed error that suggests a background job.

Query classes and expectations

The numbers below are single-instance POC measurements over 15,000,004 synthetic calls, not p95 values or SLAs. Cold is one controlled run in which private data came from object storage with no disk-cache bytes. Warm is the worst of three runs in which private data came from the NVMe cache with no object-storage bytes. Targets are proposed, not promised.

The POC corpus reused 10,000 input fixtures and small outputs. It does not represent production value cardinality or payload sizes. Loss of the NVMe cache changes latency, not correctness.

Correctness rules

• An object in S3 is immutable. The object key and checksum identify it forever. A key is a pure function of scope, stream, plane and sequence, or of scope and CID.
• A successful S3 PUT is the only upload success signal. The publisher never waits for anything else.
• A meta object's notification fires only after every data segment it references and every pack, large value and table those segments reference exist in the bucket, because the publisher uploads in that order.
• Notifications and queue delivery are at least once. Projection batch IDs, fact row IDs and Lance row IDs are deterministic from root id, relation and partition, so a duplicate produces one fact.
• An execution is projected exactly once per generation, keyed by its root id.
• A public fact for an execution is visible in ClickHouse before the Lance version that contains its private rows is committed. A query may therefore see public facts whose private rows are not yet queryable. The coordinator reports them as private_state = pending, never as NULL or a non-match. private_state is never stored in ClickHouse.
• Availability states available, not_captured, not_applicable and lost:<reason> are written by the projector as facts.
• A query binds one generation and one Lance version for its whole execution.
• Unavailable evidence is reported in queryOutcome, never silently dropped. An empty result is never produced by skipping unavailable evidence.
• ClickHouse, Lance, indexes, cache and queue are rebuildable from the bucket.
• The erasure unit is the stream, identified by its process euid, because a data segment interleaves the executions of one process and cannot be edited in place. A request to POST /v1/erasures selects process_euids, execution_ids or a time_range; execution_ids and time_range resolve to process euids first, and a selected execution erases the whole stream that contains it. Erasure removes the stream's meta segments, data segments and value packs from S3 first, then calls POST /projection-erasures with the stream's root_ids and the generation to delete its facts from ClickHouse, then commits a Lance version without its rows. Large values and function tables are shared across streams and are removed only when no remaining stream in the scope references them; that needs a reference count over the remaining streams, an open implementation decision on the system contract page. Replay cannot resurrect a deleted stream because its source is gone.

Failure behavior

Replay tool

The replay tool is an operator command, not a running service. It lists the meta objects and value packs under the evidence prefix, or reads an S3 Inventory report. It decodes each meta object's RootEnded records and compares the root ids it finds with the root ids projected in ClickHouse and in the Lance datasets. For each pack, it checks that its CIDs are in the values dataset. For each missing item it enqueues a synthetic object-created message in the same shape S3 uses, packs before meta objects. Uses:

• rebuild ClickHouse after loss, with a new projection generation;
• rebuild Lance after loss;
• recover from missed notifications; and
• replay the dead-letter queue after a fix.

POC deployment

Customer AWS account:

• one bucket with versioning on, a create-only PUT policy and optional SSE-KMS;
• evidence, derived and temporary prefixes, with lifecycle rules on temporary and an age-based rule on evidence;
• one SQS Standard queue fed by two S3 notification rules on the evidence prefix, one for the .bamlmeta.zst suffix and one for the .bamlpack.zst suffix, plus a dead-letter queue. Starting values are a 5 minute visibility timeout, 8 maximum receives and 14 day dead-letter retention;
• one gateway Lambda;
• one projector Lambda with memory sized for the largest object, 256 MiB compressed. The 64 MiB per-object cap from the earlier draft is gone;
• one scheduled index, compaction and sweep container;
• one query coordinator container on an NVMe instance with one warm replica; and
• outbound HTTPS to Boundary.

Boundary account:

• the control API;
• the PlanetScale configuration metastore;
• one projection ingest service;
• one resident query service; and
• one managed multi-tenant ClickHouse deployment.

POC acceptance tests

• One flush with all five object kinds: meta segment, data segment, value pack, large value and function table land under the expected keys, in the expected order, and the local store releases the segments.
• Retry after a lost PUT response: the next authorize answers stored for that object and nothing is uploaded twice.
• Large value stored once across two processes: the second process's authorize answers stored and the CAS key holds one object.
• Pack duplicates collapse: two packs carrying the same CID produce one row in the values dataset.
• RootEnded before its pack is ingested: the execution reports pending, the message is retried, and projection succeeds once the pack lands.
• Abandoned stream: a stream that stops without RootEnded is closed by the sweep as abandoned.
• Local checksum failure: the plane blocks, the projector quarantines exactly the executions whose data range covers the gap, and other executions of the stream continue.
• Gateway unreachable: files retained within the disk budget, flush retried.
• Malformed segment or pack with a valid checksum goes to the dead-letter queue with a quarantine marker; healthy objects continue.
• Duplicate notification, duplicate projector run and projector crash mid-execution each produce one fact set with the same IDs.
• ClickHouse outage: queue backlog, no data loss, catch-up.
• ClickHouse rebuild with a new generation, Lance rebuild, and a missed notification, all recovered by the replay tool.
• Public-only query reads no private data; private and mixed queries read Lance only within the pruned set.
• Background job: progress, cancel, downloadable result.
• NVMe node loss: same results, higher latency.
• Erasure: stream objects, facts and Lance rows gone; replay does not resurrect.
• Tenant isolation at the token, API, bucket policy and ClickHouse layers.
• Frozen aggregate ingest workload toward 50,000 executions per second across many clients. This is unvalidated. The only measured arrival replay ran in the older POC workflow: 100,000 calls over 2 seconds. It took 40.85 s until that workflow confirmed acceptance and 214.93 s to become visible.
• Frozen 13-query workload with measured cold and warm latency.