Publisher guide: shipping profiler evidence to S3

Status: Proposal, 2026-09-10. Not built. This page specifies the publisher, the component inside the BAML runtime that ships profiler files to the customer's S3 bucket. Profiler facts come from the profiler store in the BAML repository at segment schema version 2 and CAS format version 1.

This page is for the engineer who builds the publisher. It stands on its own. The hosted architecture describes the system around it, the system contract holds the OpenAPI definitions, and the storage decisions page records the bucket decisions.

What the publisher is

The publisher is a background task in the BAML runtime. It runs in the same process as the profiler, the library that records BAML calls and writes files to a local directory. Every flush interval, the publisher takes the files the profiler has closed since the last flush, compresses each one, asks the gateway for upload URLs, and PUTs the objects to S3.

flowchart LR
  PROF["Profiler<br/>closed segments and CAS files"]
  PUB["Publisher flush<br/>compress, pack, checksum"]
  GW["Gateway<br/>POST /v1/uploads/authorize"]
  S3[("Evidence bucket<br/>presigned create-only PUT")]
  Q[("Projection queue<br/>meta and pack notifications")]
  PROJ["Projector<br/>ingest packs, project RootEnded"]
  PROF --> PUB
  PUB -->|list objects| GW
  GW -->|upload_required or stored| PUB
  PUB -->|tables, values, packs, data, meta| S3
  S3 --> Q
  Q --> PROJ
  PROJ -->|read data and CAS by key| S3

Rules:

• The publisher ships closed files only. The profiler writes each file whole and renames it into place, so a file at its final path is complete.
• It ships while executions are still running. A long execution's records reach S3 in several files over time. Only the projector waits for the execution to end.
• A successful S3 PUT is the end of an upload. The publisher never calls Boundary and never waits for an acknowledgment.
• It never changes profiler bytes. Every object is one original file, compressed with zstd. Decompressing returns the exact file.
• It never decides what is sensitive. Every file goes to the customer's bucket. The projector, a worker in the customer's cloud, decides what becomes a public fact.
• Its only state is on the filesystem: a cursor file and zero-length marker files. It keeps no in-memory record of what it shipped.

Terms used on this page: the gateway is the customer-side worker that validates the publisher's token and signs S3 upload URLs. A presigned URL is a time-limited URL that permits one PUT to one key. The projector is the customer-side worker that turns objects into query data. Scope is a tenant, project and environment.

What the profiler writes

The profiler writes under <project>/.baml/profiles-v1/. The environment variable BAML_PROFILE_DIR overrides the root. Each process owns one stream, a sequence of files named by the process euid, a random 32-hex identifier.

A segment is one immutable file in a plane. A plane is the meta or data half of a stream. <seq> is a 20-digit sequence number, dense per plane, starting at 1. A missing sequence means corruption.

Two kinds of records live in a data segment group. Spans are exact records of one call: start, end, status, and references to its argument, output and error values. Only selected calls produce spans: root functions and LLM functions by default, and functions the user selects by ID. Counters are aggregates for every call, keyed by call path, in the calling context tree (CCT). Spans become calls_v1 rows; counters become call_path_stats_v1 rows.

The write cycle:

1. The profiler batches pending records in memory.
2. It writes a batch every publish_interval, default 1 second, when anything is pending. It writes sooner when pending bytes reach segment_target_bytes, which is derived from process memory and clamped between 256 KiB and 4 MiB. It also writes on flush and on engine close.
3. Each cycle writes meta records first, then data segments, then RootEnded records. A RootEnded is written only after every data group of that execution is on disk. It names the data sequence range that holds the execution.
4. Each file is written to a temp path, fsynced, renamed, and the directory is synced. Files are never appended.
5. Every file ends with a SHA-256 over all preceding bytes.

Values are content-addressed. The CID is the SHA-256 of "baml-value-v1", the codec as a big-endian u16, and the body; its string form is bamlv_1_ plus 64 hex characters. Before writing a value the store checks whether the file exists and verifies it. There is no in-memory index. A deleted file is written again on the next use. Function tables contain no program ID, so identical builds share one table across processes.

Limits the profiler enforces:

• A value larger than single_value_bytes is not written. The call records lost:value_too_large. The limit is at most 16 MiB. Nothing is truncated.
• Strings inside segments are at most 64 KiB.
• The disk budget in usage.state defaults to 10 GiB per project with 1 GiB free. baml clean removes the whole root under an exclusive lock.
• On exit, CLI paths call flush_and_join(5 s). Engine drop flushes. A crash inside publish_interval loses that batch.

What lands in S3

Every profiler file becomes one object, except small values, which travel in a value pack. A key is a pure function of scope, stream, plane and sequence, or of scope and CID.

evidence/v1/<tenant>/<project>/<environment>/
  streams/<euid>/meta/<seq>.bamlmeta.zst          notifies the projection queue
  streams/<euid>/data/<seq>.bamldata.zst          read by key
  streams/<euid>/values/<flush_ulid>.bamlpack.zst notifies the projection queue
  cas/values/<2 hex>/<cid>.bamlvalue.zst          large values
  cas/tables/<2 hex>/<cid>.bamltable.zst          function tables
  quarantine/<original key>.quarantine            written by the projector
derived/v1/<scope>/<dataset>/                     Lance datasets, internal to the data plane
temporary/v1/<scope>/jobs/<job_id>/result.ndjson  background query results

<flush_ulid> is a 26-character sortable ID the publisher assigns to each flush. <cid> is the 64-hex digest.

Key rules:

• The gateway computes every key from the authenticated scope. The publisher never chooses a key.
• Keys contain no date and no hash segment. Retention uses object age.
• Notifications fire for two suffixes only: .bamlmeta.zst and .bamlpack.zst. The projector reads every other object by key.
• A segment schema bump or a CAS format bump is a new suffix. The current suffixes are frozen.

What each object family feeds. ClickHouse is Boundary's database for public facts. Lance is the columnar dataset format in the customer's bucket for private data.

The projector reads objects with the profiler's existing stream reader, so there is no manifest for segments. The catalog states it emits are listed on the system contract.

Values: small and large

A value at or above large_value_threshold_bytes, default 32 KiB, is large. A value below it is small.

• Small values travel in a value pack, one object per flush per stream.
• Large values are one object each under cas/values/, keyed by CID. Function tables are one object each under cas/tables/. The gateway checks each CID before upload, so each is stored once per scope.

After a value or table ships, the publisher writes a zero-length marker <cid>.shipped next to it and leaves the file in place for a retention window, default 1 hour, or less under disk pressure. During the window the profiler sees the file and does not rewrite a reused value, and the publisher skips files with a marker. After the window the publisher removes both files through the store. A later reuse is written again and shipped again.

Deduplication happens in two places. Large values and tables are deduplicated at the gateway by CID before upload. Small values are deduplicated in the projector by CID when packs are ingested into values_v1. Packs in S3 may hold duplicates.

The two endpoints

Both endpoints use the PublisherToken security scheme: a bearer token bound to one scope, issued by the control API and rotated by the customer.

GET /v1/publisher/config

Called at start and every 5 minutes. The response is cacheable; its Cache-Control header starts at max-age=300.

{
  "capture_enabled": true,
  "supported_segment_versions": [2],
  "supported_cas_versions": [1],
  "flush_interval_seconds": 5,
  "max_objects_per_authorize": 100,
  "large_value_threshold_bytes": 32768,
  "max_object_bytes": 268435456,
  "pack_max_entry_bytes": 8388608,
  "shipped_value_retention_seconds": 3600,
  "upload_url_ttl_seconds": 900
}

• capture_enabled false: do not ship, keep files, poll again later.
• A segment or CAS version outside the supported lists: do not ship that file, report it.
• Every other field replaces the built-in default of the same name.
• 401 or 403: the token is invalid or does not cover the environment. Stop shipping, keep files, retry later.
• 429 or 503: back off with jitter and keep the last configuration.

POST /v1/uploads/authorize

One call lists every object the flush wants to write, up to max_objects_per_authorize. A larger flush makes several calls. The checksum is the SHA-256 of the compressed object, as hex.

{
  "process_euid": "<32 hex>",
  "flush_id": "<ulid>",
  "objects": [
    {"kind": "meta_segment", "sequence": 41, "size_bytes": 1234, "checksum_sha256": "<hex>"},
    {"kind": "data_segment", "sequence": 120, "size_bytes": 880123, "checksum_sha256": "..."},
    {"kind": "value_pack", "size_bytes": 410000, "checksum_sha256": "...", "entry_count": 812},
    {"kind": "value", "cid": "bamlv_1_<64 hex>", "codec": 1, "size_bytes": 71000, "checksum_sha256": "..."},
    {"kind": "table", "cid": "bamlv_1_<64 hex>", "codec": 2, "size_bytes": 30000, "checksum_sha256": "..."}
  ]
}

The gateway computes every key, checks each with an S3 HEAD request, and answers in the same order.

{
  "flush_id": "<ulid>",
  "objects": [
    {"kind": "meta_segment", "sequence": 41, "key": "evidence/v1/.../meta/00000000000000000041.bamlmeta.zst", "state": "upload_required", "upload_url": "...", "required_headers": {"If-None-Match": "*", "x-amz-checksum-sha256": "<base64>", "Content-Length": "1234"}, "expires_at": "..."},
    {"kind": "value", "cid": "bamlv_1_...", "key": "evidence/v1/.../cas/values/ab/ab....bamlvalue.zst", "state": "stored"}
  ]
}

The S3 PUT

The publisher sends exactly the headers in required_headers.

Timeouts: connect 5 seconds, then a body timeout of 1 MiB per second, minimum 30 seconds. A URL is valid for upload_url_ttl_seconds, default 15 minutes. A PUT that starts after expires_at needs a new authorization.

The flush algorithm

A cursor file publisher.cursor in the stream directory records the last shipped sequence per plane.

Startup:

1. Read config.
2. Open the store handle.
3. Read the committed high-water mark per plane from the store. Do not scan the directory.
4. Read publisher.cursor. If absent, ship from sequence 1. Re-shipping is idempotent.
5. Scan cas/ for value and table files without a .shipped marker. Those are pending.

A flush runs every flush_interval_seconds, or at once when pending bytes exceed pack_max_entry_bytes, or on shutdown:

1. Collect meta segments in (meta_shipped, meta_high_water], data segments in (data_shipped, data_high_water], and value and table files without a marker. Verify each file's trailing checksum. A file that fails is skipped and reported, and blocks later sequences of its plane until it validates or 60 seconds pass.
2. Compress each segment, table and large value into its own zstd frame. Build packs from the small values. Compute the SHA-256 of every object.
3. Authorize with one or more POST /v1/uploads/authorize calls.
4. Upload in this order: tables, large values, packs, data segments in ascending sequence, meta segments in ascending sequence. A PUT succeeds on 200. On 412, re-authorize the object and expect stored. On any other failure, retry that object with backoff. Never advance a plane cursor past a failed sequence.
5. After the flush's meta segments are stored: advance publisher.cursor, write .shipped markers for shipped values and tables, and release shipped segments through the store so the disk ledger is updated.
6. Remove value and table files whose marker is older than shipped_value_retention_seconds, or sooner if the store reports disk pressure.

The upload order gives the projector one guarantee: when a meta object's notification fires, every data segment it references, every pack from the same or earlier flushes of that stream, and every large value or table those data segments reference already exist in the bucket.

Shutdown: the host calls the profiler's flush_and_join, which closes the last segments, then the publisher's flush_and_join(timeout), which ships them.

Lambda: run the publisher as part of the runtime with a 1 second flush interval. Flush on the extension SHUTDOWN event, which allows about 2 seconds. Data not flushed before a hard kill is lost, as it is locally today.

Failure handling

Limits

Config changes the gateway-owned limits without a runtime release. The profiler's limits and the 100,000-entry pack cap are fixed. The values below are the built-in defaults.

At 1,000 clients and 50 executions per second each, a flush produces about one meta object, a few data objects and one pack per client, plus rare large values: 600 to 1,500 objects per second. This is arithmetic, not a measurement.

Data structures on the wire

Every object is one of the profiler's own files, compressed. This section gives the byte layout of each. Facts come from bex_prof_store (store.rs, evidence_codec.rs, cct_codec.rs, function_table.rs) and the codec-1 decoder in baml_query_profiles/src/decode.rs.

Protobuf is used in one place: the body of a codec-1 value. Everything else is a fixed binary layout the profiler already writes and reads. The publisher parses none of it beyond the headers.

Conventions

• Integers are big-endian. u16, u32, u64 are unsigned integers of that width.
• A string is u32 length followed by UTF-8 bytes.
• An optional field is one u8 presence flag (0 or 1) followed by the field when present.
• Every file starts with an 8-byte magic and a u16 format version, and ends with a 32-byte SHA-256 over every preceding byte.

Identities

Meta segment

"BAMLMET1"          8 bytes
u16 version          = 2
u64 sequence         dense per stream, starts at 1
euid[16]             must equal the directory name
u64 data_high_water  highest data sequence committed when this file was written
u64 record_count
u64 payload_len
payload              record_count records of: u8 tag ‖ u32 body_len ‖ body
sha256[32]

RootEnded is the record the projector waits for. It names the data segments that hold the execution: [data_first_seq, data_last_seq] in the same stream.

Data segment

"BAMLDAT1"          8 bytes
u16 version          = 2
u64 sequence         dense per stream, starts at 1
euid[16]
u64 group_count
u64 payload_len
payload              group_count groups, sorted by root ThreadRef
sha256[32]

One group holds one execution's records for this segment. An execution that runs across several batches has one group in each of several segments.

group:
  ThreadRef[32]            the execution (root thread)
  u8  cct_health           saturation flags
  u64 cct_record_count
  u64 cct_len
  cct_payload              counter deltas per call path
  u64 evidence_record_count
  u64 evidence_len
  evidence_payload         spans, threads, values, errors

CCT payload. Counter deltas per call path, for every call. Merged by key across the execution's segments.

u64 context_count
context_count × ( key[32] ‖ optional tuple ‖ counters )
u64 overflow_count
overflow_count × ( u8 reason ‖ u8 edge_kind ‖ counters )

tuple    = program_id[16] ‖ optional parent_key[32] ‖ u32 function_id ‖ optional call_site (4 × u32) ‖ u8 edge_kind
counters = invocations_started, spans_selected, completed_ok, completed_error, completed_cancelled,
           completed_exit, inclusive_ns (u128), direct_child_inclusive_ns (u128), await_ns (u128), await_count

The key is the SHA-256 of the tuple, so the same call path in the same program always has the same key. This feeds call_path_stats_v1.

Evidence payload. Individual facts for selected calls only.

u64 count
count × ( u8 tag ‖ u32 body_len ‖ body )

A ValueOccurrence state is either Available { cid[32], u16 codec, u64 encoded_bytes } or Lost { u8 reason }. A value is in the CAS or it is lost with a reason, for example value_too_large or queue_full.

CAS object

"BAMLVAL1"     8 bytes
u16 format      = 1
u16 codec       1 = value body, 2 = function table
u64 body_len
cid[32]         SHA-256("baml-value-v1" ‖ codec ‖ body)
body
sha256[32]      over every preceding byte

The frame is 84 bytes.

Codec 1: the value body is protobuf. It is one BamlOutboundValue message from baml_outbound.proto, the message the runtime uses to pass values across its language bridge. The trace encoder emits these variants:

Values nest: a class field or a list item is itself a BamlOutboundValue. When the engine drops part of a value, it writes a class named baml.trace.OmittedValue in its place with the reason, so the shape survives. Unknown fields are skipped under proto3 rules. The projector decodes this message into typed columns for Lance.

Codec 2: the function table body is binary.

u16 body_version = 1
u32 function_count
function_count × ( u32 function_id ‖ string fqn ‖ string display_name ‖ optional string definition_key
                   ‖ u8 kind ‖ optional string kind_detail ‖ u8 origin ‖ optional string source_file
                   ‖ optional (u32 file_id, u32 start, u32 end) ‖ optional string package_name
                   ‖ u16 namespace_count ‖ namespace strings )
u32 file_count
file_count × ( u32 file_id ‖ string path )

It feeds function_definitions_v1 and the fqn and file columns of the other relations. Source paths inside it are private and stay in the customer's bucket.

Value pack

"BAMLPCK1"        8 bytes
u16 version        = 1
u32 manifest_len
manifest           canonical JSON, uncompressed
entries            the raw CAS object files, one after another, unchanged

The manifest has sorted keys and no whitespace. It is wrapped here for reading:

{"schema":"baml-pack/1","flush_id":"<ulid>","process_euid":"<32 hex>",
 "entries":[{"cid":"bamlv_1_<64 hex>","codec":1,"offset":0,"length":812,"size":812}]}

offset and length locate an entry inside the entries region. size is the original file size and equals length. The whole pack is one zstd frame. There is no per-entry digest: each entry is a complete CAS object with its own trailing hash and CID.

What the publisher reads from these structures

1. From a segment header: the magic, the u16 version and the u64 sequence, to know what the file is and where it belongs. The trailing hash, to verify it.
2. From a CAS object header: the codec and the CID, to decide whether it is a small value for the pack, a large value, or a table.
3. Nothing else. The publisher never parses a payload, a protobuf body or a function table, and never rewrites a file.

Store changes the runtime team must make

• Add store operations: list committed sequences per plane, mark shipped, release shipped segments and expire shipped values through the usage ledger, report disk pressure. All under the existing locks.
• Populate revision_label, which is always absent today, or document that source_label is the revision identity for queries.
• Land the content-hash program_id. Today it is random per engine, so call-path identities are not comparable across processes and function_definitions_v1 emits a new row set at every engine start. This does not block the publisher.
• Provide a flush hook for hosts. Document Lambda usage: flush interval 1 second, flush on SHUTDOWN.
• Keep SCHEMA_VERSION = 2 and CAS_FORMAT_VERSION = 1 frozen for this contract. A bump is a new object suffix.

Observability

Counters and gauges:

• Objects shipped, by kind: meta_segment, data_segment, value_pack, value, table.
• Bytes shipped, compressed and original, by kind.
• Flush duration, per flush.
• Authorize latency, per call.
• Authorize outcomes: upload_required, stored, conflict, by kind.
• PUT retries and 412 responses, by kind.
• Files pending per plane, and pending value and table files.
• Age of the oldest unshipped segment, per plane.
• Bytes freed by release and by retention.

Log lines carry the object key, the plane and sequence, the flush id, the process euid, the outcome and the duration. Function names may appear in logs. Presigned URLs, value bodies and manifest contents beyond CIDs and sizes must never be logged.

Conformance checklist

An implementation must pass every test. The golden files are the profiler's fixture segments and CAS objects from the BAML repository, compressed with the reference zstd settings.

1. Given closed segments 1 to N in each plane, the publisher ships them in ascending sequence, data before meta.
2. A segment with a bad trailing checksum is not shipped. Later sequences of its plane wait. After 60 seconds the plane resumes at the next valid sequence and the failure is reported.
3. After a crash between PUT and cursor advance, restart re-ships the same objects. Every re-shipped object gets stored or 412 and the cursor advances.
4. On 412 the publisher re-authorizes that object, receives stored, and continues.
5. On 401 or 403 the publisher stops shipping, keeps every file, reloads config, and resumes when the token is valid.
6. A flush with 8 MiB plus one byte of small values produces two packs. A flush with 100,001 small values produces two packs.
7. A pack manifest is canonical JSON. Its offsets and lengths locate each entry. Each entry is an unchanged value file that verifies.
8. After a value ships, a .shipped marker exists beside it. A second flush does not ship that value again. The profiler does not rewrite that value while the marker exists.
9. After the retention window, the value and its marker are removed through the store, and usage.state reflects the freed bytes.
10. Under reported disk pressure, shipped values are removed before the window ends. Unshipped segments are never removed.
11. On the Lambda SHUTDOWN event, the publisher flushes within 2 seconds with a 1 second interval configured.
12. No data segment referenced by a RootEnded is PUT after the meta segment that holds that RootEnded.
13. Every object produced from the golden files matches the golden object bytes, and its key matches the layout on this page.
14. Config values from GET /v1/publisher/config override every built-in default, and capture_enabled false stops shipping without dropping files.
15. A conflict on a segment or pack is reported and never overwritten. A conflict on a value or table is treated as stored.

Glossary

Terms not defined in a table above.