Manage pipelines

BigQuery pipelines let you inspect run history, configure failure alerts, and control code assets powered by Dataform. You can also enrich workflow metadata in Knowledge Catalog to track data quality and delete obsolete workflows to keep your project organized. Depending on where your team stores code, you can access pipelines in Files and Folders, including user folders and team folders, or in BigQuery Studio Git repositories (Git Folders) (Preview).

Before you begin

Create a BigQuery pipeline.
To manage pipeline metadata in Knowledge Catalog, ensure that the Dataplex API is enabled in your Google Cloud project.

Required roles

To get the permissions that you need to manage pipelines, ask your administrator to grant you the following IAM roles:

To view and run pipelines:
- Dataform Viewer (roles/dataform.Viewer) on the project
- BigQuery Job User (roles/bigquery.jobUser) on the project
To run pipelines with user credentials for a Google Account: BigQuery Data Editor (roles/bigquery.dataEditor) on the project or specific BigQuery datasets
To delete pipelines: Dataform Admin (roles/dataform.Admin) on the pipeline
To manage pipelines in user folders:
- Code Owner (roles/dataform.codeOwner) on the folder
- Code Editor (roles/dataform.codeEditor) on the folder
- Code Viewer (roles/dataform.codeViewer) on the folder
To manage pipelines in team folders:
- Team Folder Owner (roles/dataform.teamFolderOwner) on the team folder
- Team Folder Contributor (roles/dataform.teamFolderContributor) on the team folder
- Team Folder Viewer (roles/dataform.teamFolderViewer) on the team folder
To manage pipelines in Git repositories: Developer Connect OAuth User (roles/developerconnect.oauthUser) on the project
To view and manage metadata in Knowledge Catalog: Dataplex Catalog Editor (roles/dataplex.catalogEditor) on the project or @bigquery entry group
To view data lineage for pipelines: Data Lineage Viewer (roles/datalineage.viewer) on the project
To open pipelines from the lineage graph: Dataform Viewer (roles/dataform.viewer) on the project that contains the pipeline

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Dataform IAM, see Control access with IAM. For more information about roles for folders, see Create and manage folders. For more information about roles for Git repositories, see Manage code with BigQuery Studio Git repositories.

If you use a custom service account to run pipelines, you must grant the following roles to that service account:

BigQuery Job User (roles/bigquery.jobUser) on the project
BigQuery Data Editor (roles/bigquery.dataEditor) on the project or specific BigQuery datasets
Dataplex Catalog Editor (roles/dataplex.catalogEditor) on the project or @bigquery entry group
Data Lineage Events Producer (roles/datalineage.producer) on the project that contains the pipeline, to report data lineage

View pipelines

You can view and inspect pipelines stored in folders or Git folders in the Files pane. You can view standalone pipelines and pipelines stored in folders in the Explorer pane. Pipelines stored in Git folders are not displayed in the Explorer pane.

View pipelines in the Files pane

Pipelines stored in folders or in Git folders appear directly in the Files pane.

To view a pipeline stored in a folder or Git folder, follow these steps:

In the Google Cloud console, go to the BigQuery page.

Go to BigQuery
In the left pane, click Files to open the file browser.

If you don't see the left pane, click Expand left pane to open the pane.
Expand your User folder, a Team folder, or a connected Git repository folder.

Pipelines are displayed with a Pipeline icon instead of a standard folder icon.
Click a pipeline folder to open the Pipeline Viewer.

The Pipeline Viewer displays the compiled Directed Acyclic Graph (DAG) of your pipeline tasks, execution status, and configuration tabs.
Expand the pipeline directory in the file browser to browse nested directories (such as definitions/) and individual task files.

View pipelines in the Explorer pane

To view a list of standalone pipelines and pipelines stored in folders, do the following:

In the Google Cloud console, go to the BigQuery page.

Go to BigQuery
In the left pane, click Explorer:

If you don't see the left pane, click Expand left pane to open the pane.
In the Explorer pane, expand your project and click Pipelines.
Select a pipeline to open it in the Pipeline Viewer.

View past manual runs

To view past manual runs of a selected pipeline, follow these steps:

In the Google Cloud console, go to the BigQuery page.

Go to BigQuery
In the left pane, click Explorer:
In the Explorer pane, expand your project, click Pipelines, and then select a pipeline.
Click Executions.
Optional: To refresh the list of past runs, click Refresh.

Configure alerts for failed pipeline runs

Each pipeline has a corresponding Dataform repository ID. Each BigQuery pipeline run is logged in Cloud Logging using the corresponding Dataform repository ID. You can use Cloud Monitoring to observe trends in Cloud Logging logs for BigQuery pipeline runs and to notify you when conditions you describe occur.

To receive alerts when a BigQuery pipeline run fails, you can create a log-based alerting policy for the corresponding Dataform repository ID. For instructions, see Configure alerts for failed workflow invocations.

To find the Dataform repository ID of your pipeline, do the following:

In the Google Cloud console, go to the BigQuery page.

Go to BigQuery
In the left pane, click Explorer:
In the Explorer pane, expand your project, click Pipelines, and then select a pipeline.
Click Settings.

The Dataform repository ID of your pipeline is displayed at the bottom of the Settings tab.

Delete a pipeline

To permanently delete a pipeline, follow these steps:

In the Google Cloud console, go to the BigQuery page. Go to BigQuery

To delete a pipeline stored in a folder or Git folder, do the following:
1. In the left pane, click Files.
2. In the file tree, find the pipeline folder that you want to delete.
3. Click View actions next to the pipeline folder, and then click Delete.
4. In the confirmation dialog, click Delete.
  
  The pipeline folder and all tasks and files contained within it are deleted from your workspace.
To delete a pipeline listed in the Explorer pane, do the following:
1. In the left pane, click Explorer:
2. In the Explorer pane, expand your project and click Pipelines.
3. Find the pipeline that you want to delete.
4. Click View actions next to the pipeline, and then click Delete.
5. Click Delete.

Manage metadata in Knowledge Catalog

Knowledge Catalog lets you store and manage metadata for pipelines. Pipelines are available in Knowledge Catalog by default, without additional configuration.

You can use Knowledge Catalog to manage pipelines in all pipeline locations. Managing pipelines in Knowledge Catalog is subject to Knowledge Catalog quotas and limits and Knowledge Catalog pricing.

Knowledge Catalog automatically retrieves the following metadata from pipelines:

Data asset name
Data asset parent
Data asset location
Data asset type
Corresponding Google Cloud project

Knowledge Catalog logs pipelines as entries with the following entry values:

System entry group: The system entry group for pipelines is @dataform. To view details of pipeline entries in Knowledge Catalog, you need to view the dataform system entry group. For instructions about how to view a list of all entries in an entry group, see View details of an entry group in the Knowledge Catalog documentation.
System entry type: The system entry type for pipelines is dataform-code-asset. To view details of pipelines,you need to view the dataform-code-asset system entry type, filter the results with an aspect-based filter, and set the type field inside dataform-code-asset aspect to WORKFLOW. Then, select an entry of the selected pipeline. For instructions about how to view details of a selected entry type, see View details of an entry type in the Knowledge Catalog documentation. For instructions about how to view details of a selected entry, see View details of an entry in the Knowledge Catalog documentation.
System aspect type: The system aspect type for pipelines is dataform-code-asset. To provide additional context to pipelines in Knowledge Catalog by annotating data pipeline entries with aspects, view the dataform-code-asset aspect type, filter the results with an aspect-based filter, and set the type field inside dataform-code-asset aspect to WORKFLOW. For instructions about how to annotate entries with aspects, see Manage aspects and enrich metadata in the Knowledge Catalog documentation.
Type: The type for data canvases is WORKFLOW. This type lets you filter pipelines in the dataform-code-asset system entry type and the dataform-code-asset aspect type by using the aspect:dataplex-types.global.dataform-code-asset.type=WORKFLOW query in an aspect-based filter.

For instructions about how to search for assets in Knowledge Catalog, see Search for data assets in Knowledge Catalog in the Knowledge Catalog documentation.

Metadata enrichment and data quality scorecard integration

Dataform can publish the following metadata to Knowledge Catalog:

Overview aspect type
Generic aspect type
Data quality scorecard aspect type

Dataform assertions are automatically integrated with the Knowledge Catalog data quality scorecard. During pipeline execution, the results of any Dataform assertions are automatically published to Knowledge Catalog. These results populate the Knowledge Catalog data quality scorecard with a pass or fail status.

To check the status of a metadata update, follow the instructions in View past manual runs.

After the metadata has been synchronized, you can search for and view the entry in Knowledge Catalog. For more information, see Search for resources.

Track data lineage

BigQuery reports table-level data lineage to Knowledge Catalog for the actions that run in your BigQuery pipeline. For each executed action, the lineage graph in BigQuery and Knowledge Catalog shows a process between the tables that the action reads and the table that it writes. You don't need to change your SQLX files to report lineage.

Tables, views, incremental tables, materialized views, assertions, property graphs, data preparations, and operations with hasOutput: true report table-to-table lineage. Operations without hasOutput: true report the process and run only. Declarations, notebooks, and SQL unit tests don't report lineage. Declarations appear only as inputs to other actions.

Reporting lineage never causes a pipeline run to fail. If the Data Lineage API is disabled, a permission is missing, or a quota is exceeded, the pipeline run continues without reporting lineage.

Data lineage ingestion is billed under the Knowledge Catalog Premium Processing SKU, and only events that write table-to-table lineage links are billed. Data lineage storage is billed under the metadata storage SKU. For more information, see Knowledge Catalog pricing.

Data lineage limitations

Data lineage for pipelines is subject to the following limitations:

Only table-level lineage is reported. Column-level lineage isn't reported.
Data lineage reporting is subject to the Data Lineage API quotas for writes per project, identity, and region.

Enable data lineage

To enable data lineage for your pipeline, follow these steps:

Enable the Data Lineage API (datalineage.googleapis.com) in the project that contains the pipeline and in the project where users view lineage.
In the project that contains the pipeline, grant the Data Lineage Events Producer role (roles/datalineage.producer) to the identity that runs the pipeline. This identity is the custom service account of the pipeline, or the Dataform service agent (service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com) if no custom service account is set.
In the workflow_settings.yaml file of your pipeline, add the following lineage block:
```
lineage:
  enabled: true
```

Pipeline runs that use end user credentials for a Google Account don't report lineage. To report lineage, run the pipeline with a custom service account.

To stop data lineage reporting for a pipeline, set enabled to false in the lineage block. To stop data lineage reporting for a whole project, disable the Data Lineage API in the project. Disabling the API also stops data lineage reporting from other services in the project.

View data lineage

To view data lineage for a table that a pipeline action produces, follow these steps:

In the Google Cloud console, go to BigQuery Studio.

Go to BigQuery Studio
In the Explorer pane, expand your project and dataset, and then select a table that a pipeline action produces.
Click the Lineage tab.
To view the action, the pipeline, and the runs, select the process node.
To open the pipeline, click Open in Dataform.

For more information about navigating lineage graphs, see View data lineage in Knowledge Catalog.

Troubleshoot data lineage

The following table describes common data lineage issues for pipelines and how to fix them:

Symptom	Likely cause	Fix
The Lineage tab doesn't show pipeline process nodes.	The Data Lineage API is disabled.	Enable the API in the project that contains the pipeline and in the project where you view lineage.
A pipeline doesn't report lineage.	The identity that runs the pipeline doesn't have the Data Lineage Events Producer role.	Grant `roles/datalineage.producer` to the custom service account or to the Dataform service agent.
Pipeline runs that use end user credentials don't report lineage.	Lineage reporting doesn't support end user credentials.	Run the pipeline with a custom service account.
A pipeline doesn't report lineage although the role is granted.	Dataform can't act as the custom service account.	Grant the Dataform service agent the Service Account Token Creator role (`roles/iam.serviceAccountTokenCreator`) on the custom service account.
One pipeline doesn't report lineage.	The pipeline is opted out.	Remove `lineage.enabled: false` from `workflow_settings.yaml`.
Lineage is missing for some runs.	The Data Lineage API quota is exceeded.	Spread out pipeline runs or request a higher quota.
Clicking Open in Dataform returns a permission error.	The user doesn't have access to the pipeline.	Grant the user the Dataform Viewer role (`roles/dataform.viewer`).

After the first permission or API error in a pipeline run, BigQuery stops reporting lineage for the rest of that run. The next pipeline run tries to report lineage again.

To check whether a pipeline run reported lineage, view the dataform.googleapis.com/lineage_emission log entries in Cloud Logging. To filter for skipped or failed lineage reporting in Logs Explorer, run the following query:

log_id("dataform.googleapis.com/lineage_emission")
jsonPayload.outcome=("SKIPPED" OR "FAILED")

What's next

Learn more about BigQuery pipelines.
Learn how to create pipelines.
Learn how to schedule pipelines.
Learn how to manage code with BigQuery Studio Git repositories.
Learn how to organize code assets with folders.
Learn more about Dataform Deployments.