Query execution across ClickHouse and Lance

The query coordinator in the customer cloud plans each SQL statement with DataFusion. Public facts come from ClickHouse in Boundary. Private columns come from Lance datasets in the customer bucket, read through a cache on local NVMe.

Coordinator opens Boundary connections Boundary never connects into the customer cloud Private predicates stay in the customer cloud

ClickHouse in Boundary: public facts

Execution and call identifiers, fqn, revision_id, status, started_at, duration_ns, the stream and data segment that hold each call, and availability states such as args_state. Public-only queries end here.

ClickHouse never holds value bodies, previews, source text or query literals that apply to private values.

Lance in the customer bucket: private columns

args, output, error, typed derived fields such as args['invoice']['id'] and an n-gram text index. They live as rebuildable Lance datasets in customer S3.

DataFusion in the coordinator reads them through a Foyer cache on local NVMe. Losing the cache changes latency, not results.

Public-onlyEvery referenced column is public. Runs in ClickHouse. Reads no Lance data.

PrivateReads a known call's values. One identity lookup, then one Lance row.

MixedPublic predicates prune in ClickHouse. Private predicates run over Lance. The coordinator joins.

BackgroundToo costly to be interactive, or requested by the caller. Returns a job ID, reports progress, writes a downloadable result.

Choose an example, then use Next step. ← →

0ClickHouse rows scanned 0Rows received from ClickHouse 0Lance rows read 0Result rows ready

Portable SQLIllustrative dataset

Inspect the DataFusion plan

The highlighted operator follows the current step. Read the logical plan from the scan at the bottom toward the result at the top.

Execution pathBoundary and Lance calls are coordinator-initiated

Client result

Inspect source rows and Lance reads Changes with each step

Dimmed rows are outside the current selection. Struck-out columns are not scanned. Private columns are highlighted only when the coordinator reads them; they never leave the customer cloud.

ClickHouse public facts

Allowlisted facts projected into Boundary.

Boundary

Lance private columns

Private rows in the customer bucket.

Customer cloud

Takeaways

Public queries never read private data. A public-only query runs entirely in ClickHouse and reads no Lance dataset.
Private predicates, such as args['invoice']['id'] = 'INV-1042', are evaluated in the customer cloud over Lance. The literal is never sent to Boundary.
Cache loss changes latency, not results. A cold read fetches Lance fragments from S3 into the NVMe cache; a warm read serves them from NVMe. Both return the same rows.
Unavailable evidence is reported in the queryOutcome. A call whose args_state is not_captured is reported in valueEvaluations.unavailable, never treated as a non-match or silently dropped. An empty result is never produced by skipping unavailable evidence.
Public facts become visible in ClickHouse before their private rows are committed to Lance. The coordinator reports such rows as private_state = pending in the privateState counts, never as NULL or a non-match.
A query binds one ClickHouse generation and one Lance version for its whole execution, and reports both in the snapshot of the outcome.