# Publisher guide: shipping profiler evidence to S3 **Status:** Proposal, 2026-09-10. Not built. This page specifies the publisher, the component inside the BAML runtime that ships profiler files to the customer's S3 bucket. Profiler facts come from the profiler store in the BAML repository at segment schema version 2 and CAS format version 1. This page is for the engineer who builds the publisher. It stands on its own. The [hosted architecture](baml-studio-final-architecture.html) describes the system around it, the [system contract](baml-control-data-plane-api.html) holds the OpenAPI definitions, and the [storage decisions page](baml-studio-s3-design.html) records the bucket decisions. ## What the publisher is The publisher is a background task in the BAML runtime. It runs in the same process as the profiler, the library that records BAML calls and writes files to a local directory. Every flush interval, the publisher takes the files the profiler has closed since the last flush, compresses each one, asks the gateway for upload URLs, and PUTs the objects to S3. ```mermaid flowchart LR PROF["Profiler
closed segments and CAS files"] PUB["Publisher flush
compress, pack, checksum"] GW["Gateway
POST /v1/uploads/authorize"] S3[("Evidence bucket
presigned create-only PUT")] Q[("Projection queue
meta and pack notifications")] PROJ["Projector
ingest packs, project RootEnded"] PROF --> PUB PUB -->|list objects| GW GW -->|upload_required or stored| PUB PUB -->|tables, values, packs, data, meta| S3 S3 --> Q Q --> PROJ PROJ -->|read data and CAS by key| S3 ``` Rules: - The publisher ships closed files only. The profiler writes each file whole and renames it into place, so a file at its final path is complete. - It ships while executions are still running. A long execution's records reach S3 in several files over time. Only the projector waits for the execution to end. - A successful S3 PUT is the end of an upload. The publisher never calls Boundary and never waits for an acknowledgment. - It never changes profiler bytes. Every object is one original file, compressed with zstd. Decompressing returns the exact file. - It never decides what is sensitive. Every file goes to the customer's bucket. The projector, a worker in the customer's cloud, decides what becomes a public fact. - Its only state is on the filesystem: a cursor file and zero-length marker files. It keeps no in-memory record of what it shipped. Terms used on this page: the gateway is the customer-side worker that validates the publisher's token and signs S3 upload URLs. A presigned URL is a time-limited URL that permits one PUT to one key. The projector is the customer-side worker that turns objects into query data. Scope is a tenant, project and environment. ## What the profiler writes The profiler writes under `/.baml/profiles-v1/`. The environment variable `BAML_PROFILE_DIR` overrides the root. Each process owns one stream, a sequence of files named by the process euid, a random 32-hex identifier. | Path | Contents | |---|---| | `streams//stream.lock` | Held while the process's store is open. Liveness signal. | | `streams//meta/.bamlmeta` | Meta segment. Records: `StreamStarted` (sequence 1 only), `EngineStarted`, `RootStarted`, `RootEnded`. | | `streams//data/.bamldata` | Data segment. One group per execution. A group holds spans for selected calls and aggregate counters for all calls. | | `cas/sha256/<2 hex>/<64 hex>.bamlvalue` | Content-addressed object. Codec 1 is a value body. Codec 2 is a function table. | | `profiles-v1.lock`, `publish.lock`, `usage.state`, `tmp/` | Locks, the disk-usage ledger, write temporaries. | A segment is one immutable file in a plane. A plane is the meta or data half of a stream. `` is a 20-digit sequence number, dense per plane, starting at 1. A missing sequence means corruption. Two kinds of records live in a data segment group. Spans are exact records of one call: start, end, status, and references to its argument, output and error values. Only selected calls produce spans: root functions and LLM functions by default, and functions the user selects by ID. Counters are aggregates for every call, keyed by call path, in the calling context tree (CCT). Spans become `calls_v1` rows; counters become `call_path_stats_v1` rows. The write cycle: 1. The profiler batches pending records in memory. 2. It writes a batch every `publish_interval`, default 1 second, when anything is pending. It writes sooner when pending bytes reach `segment_target_bytes`, which is derived from process memory and clamped between 256 KiB and 4 MiB. It also writes on flush and on engine close. 3. Each cycle writes meta records first, then data segments, then `RootEnded` records. A `RootEnded` is written only after every data group of that execution is on disk. It names the data sequence range that holds the execution. 4. Each file is written to a temp path, fsynced, renamed, and the directory is synced. Files are never appended. 5. Every file ends with a SHA-256 over all preceding bytes. Values are content-addressed. The CID is the SHA-256 of `"baml-value-v1"`, the codec as a big-endian u16, and the body; its string form is `bamlv_1_` plus 64 hex characters. Before writing a value the store checks whether the file exists and verifies it. There is no in-memory index. A deleted file is written again on the next use. Function tables contain no program ID, so identical builds share one table across processes. Limits the profiler enforces: - A value larger than `single_value_bytes` is not written. The call records `lost:value_too_large`. The limit is at most 16 MiB. Nothing is truncated. - Strings inside segments are at most 64 KiB. - The disk budget in `usage.state` defaults to 10 GiB per project with 1 GiB free. `baml clean` removes the whole root under an exclusive lock. - On exit, CLI paths call `flush_and_join(5 s)`. Engine drop flushes. A crash inside `publish_interval` loses that batch. ## What lands in S3 Every profiler file becomes one object, except small values, which travel in a value pack. A key is a pure function of scope, stream, plane and sequence, or of scope and CID. ```text evidence/v1//// streams//meta/.bamlmeta.zst notifies the projection queue streams//data/.bamldata.zst read by key streams//values/.bamlpack.zst notifies the projection queue cas/values/<2 hex>/.bamlvalue.zst large values cas/tables/<2 hex>/.bamltable.zst function tables quarantine/.quarantine written by the projector derived/v1/// Lance datasets, internal to the data plane temporary/v1//jobs//result.ndjson background query results ``` `` is a 26-character sortable ID the publisher assigns to each flush. `` is the 64-hex digest. Key rules: - The gateway computes every key from the authenticated scope. The publisher never chooses a key. - Keys contain no date and no hash segment. Retention uses object age. - Notifications fire for two suffixes only: `.bamlmeta.zst` and `.bamlpack.zst`. The projector reads every other object by key. - A segment schema bump or a CAS format bump is a new suffix. The current suffixes are frozen. What each object family feeds. ClickHouse is Boundary's database for public facts. Lance is the columnar dataset format in the customer's bucket for private data. | Family | ClickHouse | Lance | Read by the projector | |---|---|---|---| | Meta segments | `threads_v1` root rows, `health_v1` rows, execution completeness | nothing | On notification. Sequence 1 is read by key because `StreamStarted` holds the clock origin. | | Data segments | `calls_v1`, `call_path_stats_v1`, `errors_v1`, availability states, value CIDs | value references per call | By key, the range named by `RootEnded` | | Value packs | nothing | `values_v1`: cid, codec, body | On notification | | Large values | nothing | body on demand | By key from the CID | | Tables | `function_definitions_v1` public columns | source files, spans, paths | By key from `EngineStarted.function_table_cid` | The projector reads objects with the profiler's existing stream reader, so there is no manifest for segments. The catalog states it emits are listed on the [system contract](baml-control-data-plane-api.html). ## Values: small and large A value at or above `large_value_threshold_bytes`, default 32 KiB, is large. A value below it is small. - Small values travel in a value pack, one object per flush per stream. - Large values are one object each under `cas/values/`, keyed by CID. Function tables are one object each under `cas/tables/`. The gateway checks each CID before upload, so each is stored once per scope. After a value or table ships, the publisher writes a zero-length marker `.shipped` next to it and leaves the file in place for a retention window, default 1 hour, or less under disk pressure. During the window the profiler sees the file and does not rewrite a reused value, and the publisher skips files with a marker. After the window the publisher removes both files through the store. A later reuse is written again and shipped again. Deduplication happens in two places. Large values and tables are deduplicated at the gateway by CID before upload. Small values are deduplicated in the projector by CID when packs are ingested into `values_v1`. Packs in S3 may hold duplicates. ## The two endpoints Both endpoints use the `PublisherToken` security scheme: a bearer token bound to one scope, issued by the control API and rotated by the customer. ### GET /v1/publisher/config Called at start and every 5 minutes. The response is cacheable; its `Cache-Control` header starts at `max-age=300`. ```json { "capture_enabled": true, "supported_segment_versions": [2], "supported_cas_versions": [1], "flush_interval_seconds": 5, "max_objects_per_authorize": 100, "large_value_threshold_bytes": 32768, "max_object_bytes": 268435456, "pack_max_entry_bytes": 8388608, "shipped_value_retention_seconds": 3600, "upload_url_ttl_seconds": 900 } ``` - `capture_enabled` false: do not ship, keep files, poll again later. - A segment or CAS version outside the supported lists: do not ship that file, report it. - Every other field replaces the built-in default of the same name. - 401 or 403: the token is invalid or does not cover the environment. Stop shipping, keep files, retry later. - 429 or 503: back off with jitter and keep the last configuration. ### POST /v1/uploads/authorize One call lists every object the flush wants to write, up to `max_objects_per_authorize`. A larger flush makes several calls. The checksum is the SHA-256 of the compressed object, as hex. ```json { "process_euid": "<32 hex>", "flush_id": "", "objects": [ {"kind": "meta_segment", "sequence": 41, "size_bytes": 1234, "checksum_sha256": ""}, {"kind": "data_segment", "sequence": 120, "size_bytes": 880123, "checksum_sha256": "..."}, {"kind": "value_pack", "size_bytes": 410000, "checksum_sha256": "...", "entry_count": 812}, {"kind": "value", "cid": "bamlv_1_<64 hex>", "codec": 1, "size_bytes": 71000, "checksum_sha256": "..."}, {"kind": "table", "cid": "bamlv_1_<64 hex>", "codec": 2, "size_bytes": 30000, "checksum_sha256": "..."} ] } ``` The gateway computes every key, checks each with an S3 HEAD request, and answers in the same order. ```json { "flush_id": "", "objects": [ {"kind": "meta_segment", "sequence": 41, "key": "evidence/v1/.../meta/00000000000000000041.bamlmeta.zst", "state": "upload_required", "upload_url": "...", "required_headers": {"If-None-Match": "*", "x-amz-checksum-sha256": "", "Content-Length": "1234"}, "expires_at": "..."}, {"kind": "value", "cid": "bamlv_1_...", "key": "evidence/v1/.../cas/values/ab/ab....bamlvalue.zst", "state": "stored"} ] } ``` | State | Meaning | Publisher action | |---|---|---| | `upload_required` | Key is absent | PUT with the given URL and headers before `expires_at` | | `stored` | Key present with the same checksum | Treat as shipped, no PUT | | `conflict` | Key present with a different checksum | Segment or pack: report a bug, never overwrite. Value or table: treat as stored; the CID already proves the content | | Status | Meaning | Publisher action | |---|---|---| | 200 | Authorized | Continue | | 400 | Malformed request | Report a bug, drop this request, retry next flush | | 401 | Token invalid or expired | Stop shipping, keep files, reload config later, retry | | 403 | Token not allowed for this scope | Same as 401 | | 409 | The scope maps to no evidence prefix | Same as 401. Per-object mismatches never return 409; they arrive as `conflict` entries inside a 200 | | 413 | Too many entries, or one object larger than `max_object_bytes` | Too many entries: split the request, retry. Oversize object: report it, do not ship it | | 429 | Rate limited | Back off with jitter, retry | | 503, other 5xx | Gateway unavailable or failing | Back off with jitter, retry | | Network error or timeout | Gateway unreachable | Keep files, retry with backoff | ## The S3 PUT The publisher sends exactly the headers in `required_headers`. | Header | Value | Purpose | |---|---|---| | `If-None-Match` | `*` | Create-only. The PUT fails if the key exists. | | `x-amz-checksum-sha256` | base64 of the object SHA-256 | S3 verifies the compressed bytes on PUT | | `Content-Length` | compressed size in bytes | Size-bound URL | | Response | Meaning | Publisher action | |---|---|---| | 200 | Object created | Success | | 412 | Key already exists | Re-authorize that object and expect `stored` | | 403 | URL expired, or headers or body do not match | Re-authorize that object, PUT again | | 400 | Checksum mismatch | Recompute, re-authorize, PUT again. Report if it repeats | | 5xx | S3 error | Retry that object with backoff | | Response lost | Unknown outcome | Re-authorize that object. `stored` means it landed | Timeouts: connect 5 seconds, then a body timeout of 1 MiB per second, minimum 30 seconds. A URL is valid for `upload_url_ttl_seconds`, default 15 minutes. A PUT that starts after `expires_at` needs a new authorization. ## The flush algorithm A cursor file `publisher.cursor` in the stream directory records the last shipped sequence per plane. Startup: 1. Read config. 2. Open the store handle. 3. Read the committed high-water mark per plane from the store. Do not scan the directory. 4. Read `publisher.cursor`. If absent, ship from sequence 1. Re-shipping is idempotent. 5. Scan `cas/` for value and table files without a `.shipped` marker. Those are pending. A flush runs every `flush_interval_seconds`, or at once when pending bytes exceed `pack_max_entry_bytes`, or on shutdown: 1. Collect meta segments in `(meta_shipped, meta_high_water]`, data segments in `(data_shipped, data_high_water]`, and value and table files without a marker. Verify each file's trailing checksum. A file that fails is skipped and reported, and blocks later sequences of its plane until it validates or 60 seconds pass. 2. Compress each segment, table and large value into its own zstd frame. Build packs from the small values. Compute the SHA-256 of every object. 3. Authorize with one or more `POST /v1/uploads/authorize` calls. 4. Upload in this order: tables, large values, packs, data segments in ascending sequence, meta segments in ascending sequence. A PUT succeeds on 200. On 412, re-authorize the object and expect `stored`. On any other failure, retry that object with backoff. Never advance a plane cursor past a failed sequence. 5. After the flush's meta segments are stored: advance `publisher.cursor`, write `.shipped` markers for shipped values and tables, and release shipped segments through the store so the disk ledger is updated. 6. Remove value and table files whose marker is older than `shipped_value_retention_seconds`, or sooner if the store reports disk pressure. The upload order gives the projector one guarantee: when a meta object's notification fires, every data segment it references, every pack from the same or earlier flushes of that stream, and every large value or table those data segments reference already exist in the bucket. Shutdown: the host calls the profiler's `flush_and_join`, which closes the last segments, then the publisher's `flush_and_join(timeout)`, which ships them. Lambda: run the publisher as part of the runtime with a 1 second flush interval. Flush on the extension `SHUTDOWN` event, which allows about 2 seconds. Data not flushed before a hard kill is lost, as it is locally today. ## Failure handling | Condition | What the publisher does | What is at risk | |---|---|---| | Gateway unreachable | Keeps files, retries with backoff. The disk budget applies. | Freshness. Unshipped segments are never dropped before shipped ones. | | PUT fails | Retries that object with backoff | Freshness of that plane | | PUT response lost | Re-authorizes the object, expects `stored` | Nothing | | 412 on PUT | Re-authorizes, expects `stored` | Nothing | | Crash mid-flush | Cursor not advanced. Everything since the cursor re-ships. Duplicates are harmless. | Nothing | | Disk pressure | Frees shipped files first. The profiler stops recording at its budget. | New evidence, at the profiler's budget | | Token rotated | 401. Stops shipping, reloads config, retries. | Freshness until the new token loads | | `capture_enabled` false | Stops shipping, keeps files, polls config | Freshness, disk budget | | Unsupported segment or CAS version | Does not ship that file, reports it | That file and later sequences of its plane | | Clock skew | Nothing. No time appears in keys. | Nothing | | Checksum failure on a local file | Skips and reports. The plane blocks until the 60 second timeout. | The projector sees a gap and marks affected executions `index_corrupt` | | `conflict` on a segment or pack | Reports a bug, never overwrites | That object is investigated by hand | | `conflict` on a value or table | Treats as stored | Nothing | ## Limits Config changes the gateway-owned limits without a runtime release. The profiler's limits and the 100,000-entry pack cap are fixed. The values below are the built-in defaults. | Limit | Value | Owner | Reason | |---|---|---|---| | Flush interval | 5 s; 1 s on Lambda | config | Freshness against object count | | Data segment target | 256 KiB to 4 MiB | profiler | Set by the profiler from process memory | | Max value | at most 16 MiB | profiler | Larger values are lost, never written | | String in a segment | 64 KiB | profiler | Profiler cap | | Large-value threshold | 32 KiB | config | Per-object cost only where bytes justify it | | Pack | 8 MiB of entries, 100,000 entries | config, contract | Bounded pack ingest in the projector | | Objects per authorize | 100 | config | Bounded gateway HEADs per call | | Max object | 256 MiB compressed | config | An oversize single data group; one PUT, no multipart | | Shipped value retention | 1 hour | config | Avoid re-shipping hot values; bounded disk | | Presigned URL TTL | 15 minutes | config | Lambda role credential lifetime | At 1,000 clients and 50 executions per second each, a flush produces about one meta object, a few data objects and one pack per client, plus rare large values: 600 to 1,500 objects per second. This is arithmetic, not a measurement. ## Data structures on the wire Every object is one of the profiler's own files, compressed. This section gives the byte layout of each. Facts come from `bex_prof_store` (`store.rs`, `evidence_codec.rs`, `cct_codec.rs`, `function_table.rs`) and the codec-1 decoder in `baml_query_profiles/src/decode.rs`. | Structure | Exists today | Encoding | Object | |---|---|---|---| | Meta segment | yes | profiler binary, big-endian | `streams//meta/.bamlmeta.zst` | | Data segment | yes | profiler binary, big-endian | `streams//data/.bamldata.zst` | | CAS object | yes | 84-byte frame around a body; value bodies are Protocol Buffers | `cas/values/…`, `cas/tables/…`, or inside a value pack | | Value pack | new | 8-byte magic, JSON manifest, raw CAS objects | `streams//values/.bamlpack.zst` | Protobuf is used in one place: the body of a codec-1 value. Everything else is a fixed binary layout the profiler already writes and reads. The publisher parses none of it beyond the headers. ### Conventions - Integers are big-endian. `u16`, `u32`, `u64` are unsigned integers of that width. - A string is `u32 length` followed by UTF-8 bytes. - An optional field is one `u8` presence flag (0 or 1) followed by the field when present. - Every file starts with an 8-byte magic and a `u16` format version, and ends with a 32-byte SHA-256 over every preceding byte. ### Identities | Identity | Bytes | Meaning | |---|---|---| | Process euid | 16 | UUIDv7 chosen once per process. Its hex is the stream directory name. | | Engine id | `u64` | One engine instance inside the process. | | Thread id | `u64` | One logical BAML thread inside an engine. | | ThreadRef | 32 | `euid[16] ‖ engine_id ‖ thread_id`. A root ThreadRef identifies an execution. This is the `execution_id`. | | CID | 32 | SHA-256 of `"baml-value-v1" ‖ codec as u16 ‖ body`. | | Function id | `u32` | Index into the engine's function table. Not stable across recompiles. | | Program id | 16 | Random per engine today. A content hash is planned. | ### Meta segment ```text "BAMLMET1" 8 bytes u16 version = 2 u64 sequence dense per stream, starts at 1 euid[16] must equal the directory name u64 data_high_water highest data sequence committed when this file was written u64 record_count u64 payload_len payload record_count records of: u8 tag ‖ u32 body_len ‖ body sha256[32] ``` | Tag | Record | Body | When | |---|---|---|---| | 0 | StreamStarted | `u32 os_pid`, `u64 zero_unix_ns`, `string baml_version`, `string os_arch` | Record 1 of meta sequence 1, once per stream. `zero_unix_ns` is the wall-clock origin for every timestamp in the stream. | | 1 | EngineStarted | `u64 engine_id`, `program_id[16]`, optional `function_table_cid[32]`, optional `string revision_label`, optional `string source_label` | Once per engine, before its first root. `revision_label` is always absent today; `source_label` is the hex of the source snapshot hash. | | 2 | RootStarted | `ThreadRef[32]`, `u64 started_ns`, `runtime_id[16]` | An execution began. | | 3 | RootEnded | `ThreadRef[32]`, `u64 ended_ns`, `u8 status`, `u8 flags`, `u64 data_first_seq`, `u64 data_last_seq`, `u64 data_segment_count`, `u32 health_len`, `health` (26 × `u64`) | An execution finished. Status: 0 succeeded, 1 failed, 2 cancelled, 3 panicked, 4 abandoned. Flag bit 0: the RootStarted record was lost. | `RootEnded` is the record the projector waits for. It names the data segments that hold the execution: `[data_first_seq, data_last_seq]` in the same stream. ### Data segment ```text "BAMLDAT1" 8 bytes u16 version = 2 u64 sequence dense per stream, starts at 1 euid[16] u64 group_count u64 payload_len payload group_count groups, sorted by root ThreadRef sha256[32] ``` One group holds one execution's records for this segment. An execution that runs across several batches has one group in each of several segments. ```text group: ThreadRef[32] the execution (root thread) u8 cct_health saturation flags u64 cct_record_count u64 cct_len cct_payload counter deltas per call path u64 evidence_record_count u64 evidence_len evidence_payload spans, threads, values, errors ``` **CCT payload.** Counter deltas per call path, for every call. Merged by key across the execution's segments. ```text u64 context_count context_count × ( key[32] ‖ optional tuple ‖ counters ) u64 overflow_count overflow_count × ( u8 reason ‖ u8 edge_kind ‖ counters ) tuple = program_id[16] ‖ optional parent_key[32] ‖ u32 function_id ‖ optional call_site (4 × u32) ‖ u8 edge_kind counters = invocations_started, spans_selected, completed_ok, completed_error, completed_cancelled, completed_exit, inclusive_ns (u128), direct_child_inclusive_ns (u128), await_ns (u128), await_count ``` The key is the SHA-256 of the tuple, so the same call path in the same program always has the same key. This feeds `call_path_stats_v1`. **Evidence payload.** Individual facts for selected calls only. ```text u64 count count × ( u8 tag ‖ u32 body_len ‖ body ) ``` | Tag | Fact | Carries | Feeds | |---|---|---|---| | 0 | SpanStart | call ref, parent call ref, thread ref, context key, function id, call site, edge kind, `started_ns`, selection reasons, roles, optional runtime id | `calls_v1` identity and start columns | | 1 | SpanEnd | call ref, `ended_ns`, status, `inclusive_ns` | `calls_v1` end, duration, status | | 2 | SpanRuntimeId | call ref, runtime id | `calls_v1.runtime_ids` | | 3 | ValueOccurrence | call ref, context ref, role (input, output, error), state | `args_state`, `output_state`, `error_state` and the CIDs | | 4 | ErrorCapture | throw site, kind, source, value state | `errors_v1` | | 5 | TerminalErrorRef | call ref, error id | `calls_v1.error_id` | | 6 | ThreadStart | thread ref, parent, spawn call, spawn site, `started_ns`, kind, name | `threads_v1` | | 7 | ThreadEnd | thread ref, `ended_ns`, status | `threads_v1` | A ValueOccurrence state is either `Available { cid[32], u16 codec, u64 encoded_bytes }` or `Lost { u8 reason }`. A value is in the CAS or it is lost with a reason, for example `value_too_large` or `queue_full`. ### CAS object ```text "BAMLVAL1" 8 bytes u16 format = 1 u16 codec 1 = value body, 2 = function table u64 body_len cid[32] SHA-256("baml-value-v1" ‖ codec ‖ body) body sha256[32] over every preceding byte ``` The frame is 84 bytes. **Codec 1: the value body is protobuf.** It is one `BamlOutboundValue` message from `baml_outbound.proto`, the message the runtime uses to pass values across its language bridge. The trace encoder emits these variants: | Field | Type | Meaning | |---|---|---| | `null_value` (2) | empty message | null | | `string_value` (3) | string | text | | `int_value` (4) | int64 | integer | | `float_value` (5) | double | float, original bits preserved | | `bool_value` (6) | bool | boolean | | `class_value` (7) | `{ name, fields[] }` | a BAML class; `fields` are `{ key, value }` pairs | | `enum_value` (8) | `{ name, value, is_dynamic }` | an enum | | `list_value` (11) | `{ items[] }` | a list | | `map_value` (12) | `{ entries[] }` | a map of `{ key, value }` | | `media_value` (17) | `{ media, mime_type?, url or file }` | an image, audio, video or PDF reference | | `uint8array_value` (19) | bytes | raw bytes | | `bigint_value` (20) | string | a big integer as decimal text | Values nest: a class field or a list item is itself a `BamlOutboundValue`. When the engine drops part of a value, it writes a class named `baml.trace.OmittedValue` in its place with the reason, so the shape survives. Unknown fields are skipped under proto3 rules. The projector decodes this message into typed columns for Lance. **Codec 2: the function table body is binary.** ```text u16 body_version = 1 u32 function_count function_count × ( u32 function_id ‖ string fqn ‖ string display_name ‖ optional string definition_key ‖ u8 kind ‖ optional string kind_detail ‖ u8 origin ‖ optional string source_file ‖ optional (u32 file_id, u32 start, u32 end) ‖ optional string package_name ‖ u16 namespace_count ‖ namespace strings ) u32 file_count file_count × ( u32 file_id ‖ string path ) ``` It feeds `function_definitions_v1` and the `fqn` and file columns of the other relations. Source paths inside it are private and stay in the customer's bucket. ### Value pack ```text "BAMLPCK1" 8 bytes u16 version = 1 u32 manifest_len manifest canonical JSON, uncompressed entries the raw CAS object files, one after another, unchanged ``` The manifest has sorted keys and no whitespace. It is wrapped here for reading: ```json {"schema":"baml-pack/1","flush_id":"","process_euid":"<32 hex>", "entries":[{"cid":"bamlv_1_<64 hex>","codec":1,"offset":0,"length":812,"size":812}]} ``` `offset` and `length` locate an entry inside the entries region. `size` is the original file size and equals `length`. The whole pack is one zstd frame. There is no per-entry digest: each entry is a complete CAS object with its own trailing hash and CID. ### What the publisher reads from these structures 1. From a segment header: the magic, the `u16` version and the `u64` sequence, to know what the file is and where it belongs. The trailing hash, to verify it. 2. From a CAS object header: the codec and the CID, to decide whether it is a small value for the pack, a large value, or a table. 3. Nothing else. The publisher never parses a payload, a protobuf body or a function table, and never rewrites a file. ## Store changes the runtime team must make - Add store operations: list committed sequences per plane, mark shipped, release shipped segments and expire shipped values through the usage ledger, report disk pressure. All under the existing locks. - Populate `revision_label`, which is always absent today, or document that `source_label` is the revision identity for queries. - Land the content-hash `program_id`. Today it is random per engine, so call-path identities are not comparable across processes and `function_definitions_v1` emits a new row set at every engine start. This does not block the publisher. - Provide a flush hook for hosts. Document Lambda usage: flush interval 1 second, flush on `SHUTDOWN`. - Keep `SCHEMA_VERSION = 2` and `CAS_FORMAT_VERSION = 1` frozen for this contract. A bump is a new object suffix. ## Observability Counters and gauges: - Objects shipped, by kind: `meta_segment`, `data_segment`, `value_pack`, `value`, `table`. - Bytes shipped, compressed and original, by kind. - Flush duration, per flush. - Authorize latency, per call. - Authorize outcomes: `upload_required`, `stored`, `conflict`, by kind. - PUT retries and 412 responses, by kind. - Files pending per plane, and pending value and table files. - Age of the oldest unshipped segment, per plane. - Bytes freed by release and by retention. Log lines carry the object key, the plane and sequence, the flush id, the process euid, the outcome and the duration. Function names may appear in logs. Presigned URLs, value bodies and manifest contents beyond CIDs and sizes must never be logged. ## Conformance checklist An implementation must pass every test. The golden files are the profiler's fixture segments and CAS objects from the BAML repository, compressed with the reference zstd settings. 1. Given closed segments 1 to N in each plane, the publisher ships them in ascending sequence, data before meta. 2. A segment with a bad trailing checksum is not shipped. Later sequences of its plane wait. After 60 seconds the plane resumes at the next valid sequence and the failure is reported. 3. After a crash between PUT and cursor advance, restart re-ships the same objects. Every re-shipped object gets `stored` or 412 and the cursor advances. 4. On 412 the publisher re-authorizes that object, receives `stored`, and continues. 5. On 401 or 403 the publisher stops shipping, keeps every file, reloads config, and resumes when the token is valid. 6. A flush with 8 MiB plus one byte of small values produces two packs. A flush with 100,001 small values produces two packs. 7. A pack manifest is canonical JSON. Its offsets and lengths locate each entry. Each entry is an unchanged value file that verifies. 8. After a value ships, a `.shipped` marker exists beside it. A second flush does not ship that value again. The profiler does not rewrite that value while the marker exists. 9. After the retention window, the value and its marker are removed through the store, and `usage.state` reflects the freed bytes. 10. Under reported disk pressure, shipped values are removed before the window ends. Unshipped segments are never removed. 11. On the Lambda `SHUTDOWN` event, the publisher flushes within 2 seconds with a 1 second interval configured. 12. No data segment referenced by a `RootEnded` is PUT after the meta segment that holds that `RootEnded`. 13. Every object produced from the golden files matches the golden object bytes, and its key matches the layout on this page. 14. Config values from `GET /v1/publisher/config` override every built-in default, and `capture_enabled` false stops shipping without dropping files. 15. A `conflict` on a segment or pack is reported and never overwritten. A `conflict` on a value or table is treated as stored. ## Glossary Terms not defined in a table above. | Term | Meaning | |---|---| | Plane | The meta or data half of a stream | | Root | One execution: a root thread and its descendants. `RootStarted` opens it, `RootEnded` closes it. | | Selected call | A call that produces spans: root and LLM functions by default, plus functions the user selects by ID | | High-water mark | Highest sequence the store has committed in a plane | | Cursor | `publisher.cursor`, the last shipped sequence per plane | | Marker | Zero-length `.shipped` file beside a shipped value or table | | Retention window | Time a shipped value stays on disk after its marker is written | | Projection queue | SQS queue fed by notifications for meta segments and packs | | `values_v1` | The Lance dataset that holds value bodies, keyed by CID |