Skip to main content

Your first pipeline

This walkthrough takes you through the CLI's authoring loop end to end: write a pipeline.toml, compile it, validate it against the server, push it as a version, run it, and read the result. It assumes you have installed and signed in.

The shape of a pipeline​

A pipeline is authored as a pipeline.toml file. It has four top-level parts — name, nodes, edges, and optional variables:

name = "my-first-pipeline"

# One [nodes.<key>] table per node. `op` is required; `kind` and `compute_home`
# are optional (the catalog fills sensible defaults).
[nodes.data]
op = "ingest_dataset"
[nodes.data.spec]
# Everything under .spec becomes the node's config. An ingested dataset is one
# or more datawell artifacts, referenced by id — never a storage URL.
artifacts = [{ kind = "folder", uri = "<your-datawell-artifact-id>" }]

[nodes.report]
op = "export"
# .spec parameters are per-op; discover them with `blazerops catalog show export`.

# Edges wire one node's output to another's input.
[[edges]]
from = "data"
to = "report"
note

The catalog is the source of truth for which op values exist and what each node's spec accepts. List components with blazerops catalog components, and inspect one with blazerops catalog show <op>.

1. Compile (offline)​

compile turns your pipeline.toml into the pipeline IR — a build/pipeline.json file. It runs entirely offline and changes nothing on the server, so it is always safe to run:

blazerops compile

It reports the node and edge counts and a hash of the IR. A malformed file — an unknown top-level key, a node with no op, an edge naming a node that does not exist — fails here, before anything touches the server.

2. Validate (read-only)​

validate sends the compiled pipeline to the server, which checks it against the component contracts (port typing) and your organisation's policy. It is read-only — it reports problems but creates nothing:

blazerops validate

Add --coverage to see which rules were judged versus skipped.

3. Diff, then push​

Pushing creates a new version in the pipeline's lineage, so check what you are about to change first:

blazerops diff        # local pipeline vs the current server version
blazerops push        # create a new version

push is drift-guarded: if the server's current version is not the one your checkout last pushed, it refuses rather than clobber someone else's change. Override deliberately with --force.

4. Run it​

Start a run. --watch streams each node's transitions until the run reaches a terminal state; --var sets any variables the pipeline declares:

blazerops run my-first-pipeline --watch

List a pipeline's runs at any time:

blazerops runs my-first-pipeline

5. Read the result​

Every run records a bill of materials — the typed inventory of the artifacts it produced. Print it with the run id:

blazerops bom <run_id>

Recap​

You authored a pipeline.toml, compiled it offline, validated it, pushed a version, ran it, and read its bill of materials. The three authoring verbs stay separate on purpose:

  • compile is offline,
  • validate is read-only,
  • push is the only one that changes server state (it creates a version).

So compile && validate is safe to run anywhere — for example as a pull-request check — and nothing is committed until you push.

Next​