pipeline.toml

A pipeline is authored as a pipeline.toml file. blazerops compile turns it into the pipeline IR (build/pipeline.json). This page is the reference for the file format.

Top-level keys

Exactly these keys are allowed at the top level; any other fails compilation:

Key	Required	Purpose
name	yes	the pipeline's name (a string). Identifies the lineage a push versions.
description	no	free text.
nodes	yes	a table of nodes, keyed by node key (at least one).
edges	no	an array of edges wiring node outputs to inputs.
variables	no	run-scoped variables the pipeline declares.

Nodes

Each node is a [nodes.<key>] table. The table key is the node's key (used by edges).

[nodes.import_data]
op = "ingest_dataset"      # required — the component key (see `blazerops catalog components`)
kind = "transform"         # optional — the catalog default is used otherwise
compute_home = "worldgrid" # optional

[nodes.import_data.spec]    # everything here becomes the node's config.spec
artifacts = [{ kind = "folder", uri = "<datawell-artifact-id>" }]

• op is required and must be a component or recipe key from the catalog.
• kind and compute_home are optional; the catalog fills sensible defaults.
• Everything under [nodes.<key>.spec] becomes the node's config.spec. The parameters a node accepts are per-op — discover them with blazerops catalog show <op>.

caution

Data inputs are referenced by datawell artifact id, never by a storage URL. The platform resolves the id and mounts it; a raw s3://… path is not brokerable and will not run.

Edges

Edges are an array of tables. Each wires one node's output into another's input:

[[edges]]
from = "import_data"
to = "score"

Both from and to must name nodes declared in [nodes.*]. An edge to an unknown node fails compilation.

Variables

Declare run-scoped variables under [variables.<name>]. A node can reference one in its spec, and blazerops run --var <name>=<value> overrides the default for a run:

[variables.fps]
default = 2
description = "frames sampled per second"

Validation

blazerops compile is offline and fails loudly on:

• an unknown top-level key,
• a missing or non-string name,
• no nodes,
• a node with no op,
• an edge naming a node that does not exist.

Server-side checks (component contracts, port typing, org policy) run separately, and read-only, under blazerops validate.