# BAML Studio hosted architecture

> **Framework:** Sensitive values, source material and canonical profiler evidence stay in the customer's cloud. Boundary stores workflow metadata and allowlisted non-value analytical facts in its control plane. Queries use Boundary's analytical facts to select candidates and access customer storage only when exact values are required.

> **Query language:** The public query language is plain SQL as accepted by DataFusion. There is no BQL or "BAML SQL" dialect. `baml query` is the CLI command that executes SQL. The logical catalog hides the physical split between ClickHouse and customer S3; DataFusion plans the query, ClickHouse executes resident operations, and the query coordinator hydrates S3-backed columns and applies residual operations when required.

## Decisions

1. Sensitive profiler evidence stays in the customer's cloud.
2. Customer S3 is the canonical hosted evidence store.
3. Boundary operates the multi-tenant metastore and ClickHouse service.
4. ClickHouse stores only allowlisted, non-value analytical facts.
5. The customer data plane owns upload authorization, projection and value hydration.
6. Plain SQL uses the same versioned logical catalog locally and remotely.
7. The hosted query coordinator uses the existing `baml_query`/DataFusion planning contracts. ClickHouse executes resident work; the coordinator resolves customer-local data and residual value operations.
8. Work delivery is at least once. Projection is deterministic and duplicate-safe.

## Topology

```mermaid
flowchart LR
  subgraph Runtime[Customer application]
    PROF["Profiler<br/>In-process native library"]
    SPOOL["Local spool and uploader<br/>In-process background task"]
  end

  subgraph DataPlane[Customer cloud]
    GW["Gateway<br/>Alien Worker · AWS Lambda"]
    S3[("Evidence store<br/>Alien Storage · AWS S3")]
    Q[("Projection queue<br/>Alien Queue · SQS Standard")]
    WORKER["Projector + reconciler<br/>Alien Worker · SQS-triggered Lambda"]
    QUERY["Query coordinator<br/>Alien Container · EC2-backed"]
  end

  subgraph ControlPlane[Boundary cloud]
    AUTH["Auth + control API<br/>Boundary service"]
    META[("Multi-tenant metastore<br/>PlanetScale")]
    INGEST["Projection ingest API<br/>Boundary service"]
    CQUERY["Resident query API<br/>Boundary service"]
    CH[("Analytics<br/>ClickHouse Cloud")]
  end

  UI["Studio, CLI or agent<br/>Client application"]

  PROF --> SPOOL
  SPOOL -->|authorize and commit| GW
  SPOOL -->|presigned create-only PUT| S3
  GW -->|commit metadata| META
  GW -->|pointer| Q
  WORKER -.->|poll pending dispatches| META
  Q --> WORKER
  WORKER -->|read accepted bundle| S3
  WORKER -->|non-sensitive fact batches| INGEST
  INGEST --> CH
  UI -->|Boundary token| QUERY
  QUERY -->|resident subquery| CQUERY
  CQUERY --> CH
  QUERY -->|bounded exact-value hydration| S3
  AUTH --> META
```

AWS deployment mapping: Alien Workers run on Lambda. The query coordinator runs as an Alien Container on EC2-backed compute with at least one warm replica. The control plane does not initiate connections into the customer cloud; customer services call it over outbound HTTPS.

## Data placement

| Data | Location | Authority |
|---|---|---|
| Immutable profiler bundles | Customer S3 | Canonical evidence |
| Arguments, outputs, errors and captured value bodies | Customer S3 CAS | Canonical values |
| Source snapshots, source text and source paths | Customer S3 | Canonical sensitive metadata |
| Upload commitments, receipts, idempotency and projection state | Boundary metastore | Transactional workflow truth |
| Counts, timings, statuses, function names, revision IDs, token usage and value handles | Boundary ClickHouse | Rebuildable analytical projection |
| Queue messages | Customer SQS | Replaceable work delivery |

Function names and revision IDs are stored directly in ClickHouse and can be filtered, grouped and returned without customer-local lookup or obfuscation.[^function-name-sensitivity] Source code, source paths and exact captured values remain in customer S3.

ClickHouse must not contain:

- argument, output, error or log bodies;
- decoded BAML values or JSON;
- plaintext previews;
- source code or source paths;
- value-derived text, token, n-gram or vector indexes; or
- customer query literals used only for value predicates.

[^function-name-sensitivity]: V1 treats function names as non-sensitive. If that assumption changes, the architecture can add an obfuscation or customer-local mapping layer without changing where exact values are stored.

## Components

### Customer data plane

| Component | Responsibility |
|---|---|
| Gateway | Validate uploader identity, issue bounded S3 upload authorization, verify uploaded object metadata, commit the receipt to the metastore and enqueue projection work. |
| S3 evidence store | Hold immutable bundles, CAS values, source objects and anchored receipts. |
| SQS | Buffer accepted bundle pointers and drive projector retries. |
| Projector | Read an accepted bundle, validate and decode it, remove sensitive fields, and send deterministic analytical batches to the control plane. |
| Reconciler | Re-enqueue accepted commits that missed dispatch, anchor missing receipts and retry incomplete projection work. It may run in the gateway or projector process for the POC. |
| Query coordinator | Authenticate the caller, parse and plan SQL, send resident work to the control plane, hydrate values from S3, apply residual operators and return a typed outcome. |

The gateway and query coordinator may share one container for the POC. The projector and reconciler may share one worker image. They remain separate runtime roles and credentials.

### Boundary control plane

| Component | Responsibility |
|---|---|
| Control API | Tenant/project configuration, deployment registration, user and workload tokens, policy and routing. |
| Metastore service | Accepted commits, signed receipts, idempotency, dispatch state, projection generations, checkpoints and audit. PostgreSQL-compatible storage is an implementation choice; clients use HTTPS only. |
| Projection ingest API | Validate scope and projection schema, accept deterministic Arrow batches and write them to ClickHouse. |
| Resident query API | Validate a resident SQL subquery, inject tenant scope and budgets, and execute it against ClickHouse. |
| ClickHouse | Serve allowlisted analytical facts, joins, filters, grouping, ordering and rollups. |

No customer service receives direct metastore or ClickHouse credentials.

## Ingest flow

1. The uploader asks the customer gateway for an upload authorization.
2. The gateway returns a key-bound, size-bound, checksum-bound, create-only presigned S3 URL.
3. The uploader writes one sealed bundle directly to customer S3.
4. The uploader commits the uploaded bundle through the gateway.
5. The gateway verifies the S3 key, version, size and checksum without downloading the body.
6. The gateway calls the Boundary metastore API. One transaction records the accepted commit and its projection outbox item.
7. The metastore returns a signed receipt. The gateway anchors it in customer S3 and returns it to the uploader. The uploader can reclaim its local spool only after this receipt.
8. The gateway immediately sends a pointer to customer SQS and records dispatch with the metastore.
9. If the immediate send is missed or ambiguous, the reconciler reads pending outbox items from the metastore and sends them again.
10. A projector reads the pointer, re-checks accepted state, fetches the bundle from S3, validates it and produces deterministic non-sensitive fact batches.
11. The projector sends the batches to the Boundary projection ingest API and completes the commit checkpoint.
12. The metastore advances the queryable watermark only after every required batch is visible.

The metastore commit and SQS send are not one transaction. The durable outbox item closes that failure gap. A separate outbox deployment is not required.

## Query flow

1. Studio, the CLI or an agent receives a short-lived Boundary token scoped to tenant, project and environment.
2. The client calls the query coordinator in the customer data plane.
3. The coordinator validates the token and binds a projection generation and projected-through watermark.
4. `baml_query` parses and plans the SQL against the versioned logical catalog.
5. The coordinator sends the safe resident subquery, including function-name and revision predicates, to the Boundary resident query API.
6. ClickHouse performs resident filtering, joins, grouping, ordering and candidate reduction.
7. If the query references exact values, ClickHouse returns bounded candidates with authorized value handles.
8. The coordinator batch-reads the required values from customer S3 and applies residual value predicates and projections.
9. The coordinator returns rows and the mandatory typed query outcome.

Resident-only queries do not read S3. When a `WHERE`, `ORDER BY` or other operation depends on an S3-backed value, the coordinator must evaluate that operation before applying the query's final `ORDER BY` and `LIMIT`. It may page bounded candidate sets from ClickHouse, hydrate their values and stop when the result is complete. It must not apply `LIMIT` to ClickHouse candidates first if doing so could change which rows satisfy the query. Queries that would require an unbounded S3 value scan are rejected or routed to a separate export workload. This rule concerns SQL execution order across ClickHouse and S3; it is unrelated to ClickHouse deduplication or `LIMIT 1 BY`.

> **Pending design work: S3 predicate scans.** Define and benchmark how the coordinator pages ClickHouse candidates and batches S3 value reads for predicates over exact values. The design must avoid one serial S3 request per candidate, bound concurrent requests and hydrated bytes, support cancellation, and define caching and short-circuit behavior. The POC must measure this path separately from resident-only ClickHouse queries.

## Correctness rules

- Accepted evidence is immutable.
- An uploaded object is not accepted until the metastore commits it and issues a receipt.
- The same commit ID and manifest digest is idempotent success.
- The same commit ID with different immutable fields is a conflict.
- SQS and projection delivery are at least once.
- Projection batch IDs and row IDs are deterministic.
- Duplicate batches produce one semantic fact.
- A hash conflict for the same logical identity is quarantined; latest arrival does not win.
- ClickHouse and SQS can be rebuilt from accepted S3 evidence plus metastore state.
- A query binds one projection generation and watermark.
- Unavailable evidence is reported in the query outcome, not treated as `NULL` or a non-match.

## Failure behavior

| Failure | Behavior |
|---|---|
| Control plane unavailable during upload | The object may remain in S3, but it is not accepted. The uploader retains its local spool and retries commit. |
| Metastore commit succeeds but SQS send fails | The reconciler republishes the committed outbox item. |
| SQS send succeeds but dispatch acknowledgement fails | The pointer may be delivered twice; projection remains idempotent. |
| Projector crashes | SQS redelivers after the visibility timeout. |
| ClickHouse is unavailable | SQS buffers work; accepted evidence remains safe in S3. |
| ClickHouse is lost | Recreate the active generation and replay accepted evidence. |
| Customer S3 value is missing or denied | Return an incomplete typed outcome and record an evidence issue. |

## POC deployment

Customer AWS account:

- one S3 bucket or isolated prefix;
- one SQS Standard queue and dead-letter queue;
- one gateway/query container;
- one projector/reconciler worker; and
- outbound HTTPS access to Boundary APIs.

Boundary account:

- one multi-tenant metastore service;
- one projection ingest service;
- one resident query service; and
- one managed multi-tenant ClickHouse deployment.

## POC acceptance tests

- Upload, commit, receipt and local spool reclamation.
- Metastore commit followed by failed SQS send.
- Duplicate SQS delivery and duplicate projection batch.
- Projector crash and retry.
- ClickHouse outage, backlog and recovery.
- ClickHouse rebuild from S3 plus metastore state.
- Resident-only query without S3 access.
- Query with bounded S3 value hydration and residual filtering.
- Rejection of an unbounded value scan.
- Tenant isolation at the token, API and ClickHouse layers.
- A frozen 50,000-executions/s ingest workload with measured bundle rate, bytes per second and projection lag.
- A separate frozen dashboard query workload with measured p95 and p99 latency.