Harness, kernel, and config

Every component in a pipeline is three separable layers. Keeping them separate is what lets the platform absorb a problem nobody designed for in advance.

Layer	Owns	Written by	Changes
Config	topology and scale — params, variable references, parallelism, sample, target, image	the user or the agent	freely, without touching code
Harness	mechanism — the loop, sharding, concurrency, IO, progress, cancellation, checkpoints	the platform	never
Kernel	semantics — what one unit of work means	whoever authors the node	with every new problem

The rule that makes this work: each layer's owner never writes the layer below.

Why the split exists

Semantics are unbounded. There is no limit to what "one unit of work" might mean — a frame to classify, a scene to reproject, a document to extract from, a training step.

Mechanism is tiny. A loop is one of three shapes (map, expand, reduce), parallelism is some number of workers times some concurrency per worker, and IO is "read the unit, write the result." That is nearly the whole space.

So a problem the platform has never seen needs new semantics only. And the plumbing that is written badly, non-deterministically, one authoring loop at a time, is not merely discouraged: it is structurally unavailable. A kernel cannot get sharding wrong if it cannot express sharding.

Declare, then derive

A kernel declares intent; the platform derives mechanism from the declaration. A kernel declares a SHAPE alongside its logic, and the platform reads that declaration to decide how to run it:

• SHAPE.exports names run-scoped values the kernel produces — the platform lifts them into the run's variable ledger with provenance.
• mode: reduce means the node sees the whole set at once, so the platform pins it sequential and never fans it.
• out.columns is the data contract the kernel promises to emit — checked at smoke, so a mismatch surfaces before it reaches a CSV.
• Declarations about parallelism safety and network need decide the dispatch posture.

Every declaration the platform derives plumbing from is an entire class of bug deleted.

The governing invariant

Everything a kernel can express must be either enforced by the harness or rejected by it — loudly, at smoke. Expressible-but-silently-ignored is the only forbidden state.

A kernel that quietly does something the platform does not support is the dangerous case, because it passes a small test and then corrupts a corpus run. Four guard layers uphold this:

Guard	When	What it does
Expressibility	as code is written	rejects a body with no kernel entry point; refuses concurrency primitives, process-group setup, raw network clients, and credential reads
Declaration lift	at registration	parses declarations and derives plumbing from them; something needed but undeclared fails at dispatch rather than mid-run
Loud refusal	at smoke	a kernel reaching for harness-owned mechanism aborts with a machine-readable code, not a warning
Smoke matches corpus	always	the root guard — the smoke test runs the same code path, the same image, and the same input contract as the corpus run

The last one carries the most weight. Nearly every "it looked fine and wasn't supported" failure traces back to smoke and corpus differing somewhere. You can run a kernel through this exact harness locally with blazerops kernel test.

Three planes of information

Information moving through a pipeline belongs to exactly one plane, and the boundaries are what stop a pipeline from turning into a notebook full of hidden globals.

Plane	Carries	Granularity	Rule
Data	per-unit facts — rows, frames, timestamps	one value per unit	a column, stamped once at the producer
Variables	run-scoped scalars — a sampling interval, a class count, a threshold	one value per run	declared, single producer, never copied
Progress	how far along an attempt is	per node attempt	emitted by the harness only, never an input

A value that every unit has its own copy of belongs in the data plane; putting it in variables recreates global-variable soup. A value the whole run shares belongs in the variables plane; copying it into three nodes' params is how the same number silently diverges into three different numbers.

Variables have one producer — either the run's seed or exactly one node. A second producer is a validation error when the pipeline is built, not a surprise when it runs. Because a variable's resolved value is part of what identifies a node's work, changing a variable correctly produces a different result rather than a stale cached one.

The kernel's whole world: ctx

A kernel receives one unit and a context object. The context is the harness, and it is the only surface the kernel has.

Surface	Provides
ctx.params	parameters with variable references already resolved to concrete values
ctx.export(name, value)	emit a run-scoped variable with provenance
ctx.emit(...) / ctx.emit_row({...})	emit data-plane output; the harness handles indexing and naming
unit iteration	local files, one unit at a time — no URIs, no credentials, no sampling logic
ctx.tools.*	brokered calls to language models and HTTP services, with keys held server-side
ctx.checkpoint(state) / ctx.resume_state	durable resume across a stop or failure
ctx.rank / ctx.world_size	shard identity, for the rare kernel that genuinely needs it

The harness is never generated. It ships inside the worker image and is versioned into the identity of the work, so a harness bug is fixed once, in one place, for every pipeline at once.