Skip to content
Merulan

Run a data pipeline

Start a Pipeline path in the Console and use Data Pipelines for the workflow and its readback.

Use this guide when you need to run data through a Pipeline and observe the result. Connect and map a Source owns the first path. Promote and troubleshoot owns execution recovery. Runs and lineage owns the readback. This guide only connects the Console to those pages.

The four steps of running a data pipeline

The path of the task: create a Source, create a Pipeline, promote a Run, then read the Run.

Before you start

The Runs and lineage page names the result to keep after the work begins. Do not replace that result with a guide-specific record. Choose the path after you confirm the input belongs in this Workspace.

Open Data Pipelines

The Console is the browser Surface for this journey. Open it before you start the Data Pipelines workflow.

  1. In the Console, open Data Pipelines.
  2. Start the Connect and map a Source workflow for your input.
  3. Keep the result that Runs and lineage tells you to read back.

Read the result

Read the result in the Data Pipelines location named by Runs and lineage. Success: the result has a visible state and readback in that location. That page explains what the state means and how to continue. That readback is the shared observation for the Console and every other Surface. Use that state before you choose the next step.

When it fails

Access denied

Cause: the current Workspace cannot start the Data Pipelines workflow. Recovery: use Administer workspace access to resolve membership or permission, then return to the Console.

The Console does not offer the action

Cause: this Workspace does not reach the Data Pipelines operation you need. Recovery: use Promote and troubleshoot to choose the supported recovery for the Run state.

Next steps