Customize Salesforce data ingestion

Important

Microsoft is making Nonprofit data solutions in Microsoft Fabric available as open-source assets so partners and the wider nonprofit community can adapt and build on them over time. Beginning February 1, 2027, Microsoft will no longer provide product engineering support or guarantee future maintenance, updates, or compatibility. As part of this transition, the Microsoft-published Nonprofit data solutions offer will be removed from Microsoft Fabric. Assets you already deployed remain in your workspaces, and you can continue to use them.

The Microsoft-published Nonprofit data solutions offer will be removed from Fabric, so you won't be able to obtain it there after removal. A partner-led solution might become available in the future, but Microsoft can't guarantee an offer, timing, or availability for every customer or market. If you currently use this solution, confirm which assets are already deployed in your workspace and that they continue to work in your environment. Those deployed assets remain in your workspace and you can continue to use them after the Microsoft-published offer is removed from Fabric. Document integrations, dependencies, security needs, and support ownership, and monitor future guidance.

Partners should use the open-source assets and may publish their own solutions in Microsoft Fabric after completing the applicable publishing requirements for the Fabric Workload Hub. Partners may use the open-source assets to build and support their own solutions. Before presenting an offer as available in the Microsoft Fabric Workload Hub, partners must complete applicable Microsoft Fabric publishing requirements and clearly define their support, security, pricing, capacity, and escalation model.

Important

Some or all of this functionality is available as part of a preview release. The content and the functionality are subject to change.

You can customize Salesforce data ingestion by adding new tables from Salesforce or by adding new columns to existing tables. This article provides guidance on both scenarios.

Add new tables from Salesforce

To add new tables, you must modify the ingestion pipeline and update the merge logic.

Modify the ingestion pipeline

  1. Open the Fundraising_SalesforceNPSP_BR_Load pipeline. This pipeline handles the incremental loading of Salesforce data into the bronze layer.

    Screenshot of the pipeline for ingesting Salesforce tables

  2. Add a new Copy data activity for each extra table.

    • Follow the same structure used for existing tables.
    • Update the source, destination, and mapping for the newly added Copy Activity.
    • Ensure that the source and sink datasets are properly configured.

    Copy data activity

  3. Insert a Filter activity before each Copy data activity to ensure the watermark of the last update for this table is used for comparison.

    Add a filter activity before a Copy data activity

Update the merge logic

  1. Modify the Fundraising_SalesforceNPSP_BR_Merge notebook.

  2. Add the new table names to the main_table_names array so that you can allow merging from staging into the final bronze tables.

    Add new table names to the array

Add new columns to existing tables

To ingest more columns into existing Salesforce tables in the bronze lakehouse, follow these steps:

  1. Open the Fundraising_SalesforceNPSP_BR_Load pipeline. This pipeline handles the incremental loading of Salesforce data into the bronze layer.

    Screenshot of the pipeline for ingesting Salesforce tables

  2. Edit the existing Copy data activity.

    a. Open the Copy data activity.
    b. Navigate to the Source tab.
    c. Edit the SOQL query by adding the desired column.

  3. Edit the mapping.

    a. Navigate to Mapping tab and select Import schemas.

    Screenshot of the Import schemas option

    b. To run the Import mapping command, use example values as shown in the following image:

    Example values for the Import mapping option

  4. Save the data pipeline. With the next pipeline run, the merge process automatically adds the new column to the corresponding table.

Next steps

You can now process your data through the transformation and enrichment pipelines.

Transform data to the silver layer

To make your new data available in the silver layer:

  1. Extend the pipeline in the bronze layer to the silver layer to set up the transformation pipeline.
  2. Customize or extend the silver data model to create new tables or modify the silver layer schema.
  3. Customize the transformation logic to define how your Salesforce data is transformed.

Optionally, you can customize the configuration logic to add custom configuration settings for your transformations.

Enrich data to the gold layer

To make your transformed data available for reporting:

  1. Extend the pipeline in the silver layer to the gold layer to set up the enrichment pipeline.
  2. Customize or extend the gold data model to create new analytical tables or modify the gold layer schema.
  3. Customize the enrichment logic to create custom calculations and metrics.

Optionally, you can customize the segmentation logic to define custom constituent segments based on your organization's needs.

See also

  • Last updated on