Pipelines
Pipelines define how input becomes Records in Data Pipelines. A Pipeline owns a graph of Sources, transforms, mappings, checks, and output targets. Each saved version keeps that graph. A Run executes one version and records what happened.
Ownership
A Pipeline belongs to a Workspace. Its source nodes name the inputs. Its mapping connects input columns to fields in the Company Schema. The target names the Entity or Relationship kind, identity rule, and write policy. Sources and media owns connection and review evidence. The Pipeline owns the rules that turn that input into proposed or written values.
Lifecycle
A Pipeline keeps immutable versions. Editing the graph creates a later version. The retained Run keeps the version it executed. Validation checks the graph, mapping, and targets before execution. A dry run shapes input and keeps evidence without promoting Records. A separate promotion Run applies the target's write policy after its checks pass.
The Pipeline's graph can combine inputs and transform rows before mapping. The accepted graph and current Source bindings determine whether a Run can proceed. A graph that can be displayed is not proof that each node can execute.
Identity and writes
A Pipeline target uses identity to find the Record a row refers to. The chosen strategy can use a semantic key, an external id, or a hash of an input span. Identity and write policy answer separate questions: which Record, then which change. Check both before starting a Run.
| Write policy | Result |
|---|---|
upsert | Updates a matching Record and creates unmatched Records. |
append | Declares that a target only adds. Promotion still resolves a row to a matching Record when identity finds one. |
candidate_review | Stages rows for a decision instead of promoting them. Merulan promotes a staged row only where a review path accepts it. |
Boundaries
A Pipeline validates against the installed Company Schema. A target can pin the Entity or Relationship revision it was authored against. A later schema change can make that target stale. Passing an earlier check does not grant permission to write against a changed definition.
Pipeline validation also differs from a standing Constraint. Pipeline checks govern the rows and outputs in this execution. Quality checks assess evidence over the scope their Constraint declares.
Failures
A Pipeline target or node diagnostic names the graph node, target, field, or mapping that cannot proceed. The code target_kind_version_stale means the target pins no revision of its Entity or Relationship definition, or pins a version other than the installed one. The code target_missing_evidence means the target does not declare evidence.required = true. The code unsupported_node_kind means the graph names a node the planner cannot run. Correct the named cause before promotion. A readable graph alone cannot prove that its output is valid.
Recovery
A Pipeline repair belongs in a later version when its mapping or target is wrong. Recheck the current schema, revise the affected field or target, then validate and preview that version. Keep the failed Run to explain the earlier result. Follow Promote and troubleshoot when the mapping is ready and execution needs repair or readback.
Related
- Connect and map a Source builds a checked path from input to a Run.
- Runs and lineage explains the status, artifacts, and history a Pipeline execution keeps.
- Company Schema owns the definitions a Pipeline target must match.