Skip to main content

Components and nodes

A pipeline is a typed directed graph. A component is one node in it: a declared contract plus a body of work. The component declares what it needs and what it produces; the platform supplies everything in between.

Read Harness, kernel, and config first — a component is where those three layers meet.

A component declares, it does not orchestrate​

What a component declares in its configuration:

  • params — its knobs, which may reference run-scoped variables rather than hard-coding values
  • parallelism — the shape of its work (per-cell, data-parallel, distributed, single) and the quantity (how many workers), kept deliberately separate
  • image — the capability profile its code needs
  • target — a placement hint, not a choice of infrastructure
  • network — whether it needs egress, which defaults to denied

What it never declares: a queue, an engine, a node group, a credential, or a file path.

Routing follows the output type​

A component's execution plane is derived from what it produces, not from a setting someone remembered to flip. A component that outputs a model trains. One that outputs data derives. One that targets a deployed model infers. Placement is a hint within a plane and never decides the plane itself.

Three kinds of body​

A component's actual work comes from one of three places:

  • A builtin operation — platform code for a well-defined transform.
  • A template — platform-authored, versioned code the component names, for recurring shapes like calling a language model over a set of items.
  • An authored kernel — code written for this specific node.

All three run through the same harness and honour the same contract. The difference is only who wrote the body.

The authoring surface​

An authored kernel is a per-unit function and a declaration. That is the entire surface:

SHAPE = {
    "mode":    "map",                  # map | expand | reduce
    "unit":    "frame",                # what one call processes
    "reads":   ["sample_every_s"],     # run-scoped variables consumed
    "exports": ["class_count"],        # run-scoped variables this kernel produces
    "in":      {"columns": ["timestamp_sec", "modality"]},
    "out":     {"columns": ["timestamp_sec", "start", "end"]},
}

def process(item, ctx):
    ...                                # domain logic only

SHAPE is machine-readable intent. The declared columns become a contract checked when the kernel is smoke-tested, and they flow into build-time validation, where an upstream node's declared output must cover what a downstream node declares it reads. A mismatch fails when the pipeline is built, with a precise message — not silently, in a file, later.

The three modes are the whole vocabulary of mechanism: map is one result per unit, expand is many results per unit, reduce sees the set. Everything else — how many workers, how the work is sharded, how output is named and indexed — is the harness's business.

Authoring is part of the node, not a node of its own​

A node that needs code carries the instruction to write it. When the pipeline is built, that node authors its own body, proves it on sample data, and pins the result. The pin is what makes the node deterministic from then on: later runs reuse the pinned code rather than re-authoring, and a retry re-executes rather than re-writing.

This keeps the graph honest. Edges carry data, and code is a property of the node that produced it — not a second kind of edge threading a code reference between boxes.

The catalog is configuration​

The set of available components, their ports, and their types is declarative configuration the platform serves — not a hard-coded list. From the CLI, list them with blazerops catalog components and inspect one with blazerops catalog show <op>.

Ports are typed, and edges are type-checked before a pipeline is allowed to run: an output must satisfy the input it feeds.

Everything a component produces is accounted for​

Each artifact a run produces folds into one of four classes — data, code, weights, or an analytic result — and each class has a storage authority that owns it. A run therefore has a complete bill of materials describing what was produced, by which component, from which inputs. That record is what makes a result auditable rather than merely present. Print it with blazerops bom <run_id>.