Extend data pipelines from one layer to another

Important

Some or all of this functionality is available as part of a preview release. The content and the functionality are subject to change.

You can extend data pipelines from one layer of the medallion architecture to another. Based on your requirements, you can extend pipelines from the bronze (ingestion) layer to the silver (transformation) layer, or from the silver layer to the gold (enrichment) layer.

This article provides guidance on how to customize and extend the data transformation and enrichment pipelines, including modifying the data model, configuration logic, transformation logic, enrichment, and segmentation.

Extend pipeline in the bronze layer to the silver layer

The bronze-to-silver transformation pipeline is responsible for creating the silver data model, configuring transformations, and transforming data from the bronze data lake to silver tables.

  1. Sign in to Microsoft Fabric.

  2. In the left navigation pane, select Workspaces and navigate to the workspace where Nonprofit data solutions is installed.

  3. Locate and open the Fundraising_BR_Ingestion pipeline, responsible for transforming data from the bronze data lake to tables in the silver layer.

    Screenshot of the pipeline for transforming data from bronze data lake to silver layer.

Customize or extend the silver data model

To customize or extend the silver data model:

  1. Create a new Notebook activity and attach a notebook with your custom code to it.

  2. Place it after the last notebook related to the silver data model creation or extension.

  3. Connect the activity with the rest of the activities.

  4. Select Save.

    Screenshot of the pipeline with a new activity added.

Customize the configuration logic

To customize the configuration logic or create a new configuration:

  1. Create a new Notebook activity and attach the notebook with your custom code to it.
  2. Place it after the last notebook related to configuration creation.
  3. Connect the activity with the rest of the activities.
  4. Select Save.

Customize the transformation logic

To customize the transformation logic or create custom transformations for new data sources, you have two primary options depending on the complexity of your changes.

For minor adjustments, such as adding columns or modifying column mappings, you can directly edit the existing transformation notebooks:

  • Fundraising_D365_Transform: Use this notebook to customize transformations for Dynamics 365 data.
  • Fundraising_SalesforceNPSP_Transform: Use this notebook to customize transformations for Salesforce NPSP data.

Common customizations include:

  • Mapping changes: Change which source column maps to which target column in the silver layer.
  • Adding columns: Include extra columns from the source data that weren't originally mapped.

For more complex requirements, such as custom unification or validation logic specific to your business needs, we recommend adding a new notebook to the pipeline.

  1. Create a new Notebook activity and attach a notebook with your custom code to it.
  2. Place it after the last notebook responsible for transformation in the Fundraising_BR_Ingestion pipeline.
  3. Connect the activity with the rest of the activities.
  4. Select Save.

This approach lets you implement specific business rules without modifying the core transformation logic provided by the solution.

Extend pipeline in the silver layer to the gold layer

The silver-to-gold enrichment pipeline is responsible for creating the gold data model, performing enrichment, and creating segmentation.

  1. Sign in to Microsoft Fabric.

  2. In the left navigation pane, select Workspaces and navigate to the workspace where Nonprofit data solutions is installed.

  3. Locate and open the Fundraising_SL_GD_Enrichment pipeline, responsible for creating the gold data model, enrichment, and segmentation.

    Screenshot of the pipeline for transforming data from silver layer to gold layer.

Customize or extend the gold data model

To customize or extend the gold data model:

  1. Create a new Notebook activity and attach a notebook with your custom code to it.

  2. Place it after the last notebook related to the gold data model creation or extension.

  3. Connect the activity with the rest of the activities.

  4. Select Save.

    Screenshot of the pipeline with a new activity added for the transformation to gold layer.

Customize enrichment logic

To customize the enrichment logic or create custom enrichment for new data sources:

  1. Create a new Notebook activity and attach the notebook with your custom code to it.
  2. Place it after the last notebook responsible for enrichment.
  3. Connect it with the rest of the activities.
  4. Select Save.

For detailed guidance on customizing enrichment, see Customize enrichment insights.

Customize segmentation logic

To customize the segmentation logic or create custom segmentation for new data sources:

  1. Create a new Notebook activity and attach the notebook with your custom code to it.
  2. Place it after the last notebook responsible for segmentation.
  3. Connect it with the rest of the activities.
  4. Select Save.

For detailed guidance on customizing segmentation, see Customize segmentation logic.

Deactivate an activity

Important

We recommend that you deactivate activities in the pipelines instead of deleting them. This approach helps preserve the pipeline structure and makes it easier to troubleshoot issues.

Follow these steps to deactivate an activity from a pipeline in the Fabric workspace:

  1. Open a pipeline and select an activity you want to delete.
  2. Select the General tab.
  3. In the Activity state field, select Deactivated.
  4. In the top menu bar, select Save.

See also

  • Last updated on