Skip to main content

Caching and reuse

Re-running a pipeline should redo only what actually changed. Reuse is how that happens, and it rests on one idea: work is identified by its content, not by when it ran.

How work is identified​

Three hashes chain together:

  1. A node's output is fingerprinted from the bytes it produced.
  2. That fingerprint feeds the input identity of everything downstream.
  3. Combined with the node's own identity — its configuration, its resolved parameters, and its code — this gives each node a hash that fully describes the work it represents.

Change one node and the change propagates down that path only. Branches that did not change keep their identity and are reused.

The invariant​

A run that reuses cached work is byte-identical to one that does not.

Reuse is only ever an optimization. If a cached run and an uncached run can differ, the cache is wrong — the run is not.

What invalidates reuse​

  • Changed configuration or parameters. Identity is computed over the resolved specification, so a run-scoped variable that changes value correctly produces a different identity and recomputes.
  • Changed code. A kernel's content is part of its node's identity, so editing a body invalidates it and everything downstream.
  • A new template version. Platform-authored bodies carry a version, so improving one invalidates work that used the old one instead of silently serving stale results.

Build and run never share results​

A build runs the pipeline over a sample; a run executes it over the full corpus. These must be distinguishable, or a full run would happily reuse the build's small sample output and quietly produce a fraction of the expected result. The sample is therefore part of a node's identity: a build's output and a run's output are different work, get different identities, and never satisfy each other.

Reuse is also how resume works​

Because a built node is deterministic — its code is pinned, so its identity is stable — a failed or stopped pipeline does not restart from zero. Every node whose identity is unchanged is reused, and execution picks up from the first node that changed or failed. Rebuilding a single node busts only that node's identity, so it and its downstream recompute while everything upstream is reused.

This is what turns reuse from a performance trick into the mechanism behind stop, fix, and resume.

Controlling it​

A run reuses cached work by default. To force a full recomputation — recomputing every node regardless of whether its identity is unchanged — pass --no-cache:

blazerops run my-pipeline --no-cache

Results are always stored regardless, so a run leaves behind everything a later run could reuse.