Connect additional data sources

Important

Some or all of this functionality is available as part of a preview release. The content and the functionality are subject to change.

Nonprofit data solutions in Microsoft Fabric supports connecting data from multiple sources beyond Salesforce and Dynamics 365. You can bring data from any data source supported by Microsoft Fabric into the bronze layer and integrate it with your data pipelines.

Prerequisites

Before connecting additional data sources, ensure you have:

  • A Microsoft Fabric workspace with Nonprofit data solutions installed
  • Appropriate permissions to create and modify pipelines
  • A configured connection to your data source in Microsoft Fabric

Create an ingestion pipeline for a new data source

  1. Sign in to Microsoft Fabric.

  2. In the left navigation pane, select Workspaces and navigate to the workspace where Nonprofit data solutions is installed.

  3. Create a new data pipeline for your data source by following these guidelines:

    • Create a separate bronze lakehouse for each data connector to maintain data isolation.
    • Configure incremental loading using watermarks to track the last update timestamp.
    • Use Copy data activities to transfer data from your source to the bronze layer.
    • Implement the merge logic in a notebook to consolidate the staging data into the final bronze tables.

  4. You can review the Fundraising_SalesforceNPSP_BR_Load pipeline as a reference to understand the structure and inspect all activities and notebooks.

Integrate the new pipeline

After creating your ingestion pipeline for the new data source:

  1. Locate and open the Fundraising_BR_Ingestion pipeline, which orchestrates all the ingestion pipelines.
  2. Create a new Invoke pipeline activity and add your newly created ingestion pipeline.
  3. Connect the activity with the last activity in the list, and select Save.

Best practices

When connecting additional data sources, consider the following best practices:

  • Data isolation: Create a separate bronze lakehouse for each data connector to maintain clear data boundaries.
  • Incremental loading: Implement watermark-based incremental loading to optimize performance and reduce data transfers.
  • Error handling: Add appropriate error handling and logging to your pipelines to troubleshoot issues.
  • Data quality: Validate data quality at the ingestion layer before moving to the transformation layer.
  • Documentation: Document the schema and data mappings for your new data source for future reference.

Next steps

After successfully connecting your additional data source to the bronze layer, you can extend the data pipelines to transform and enrich your new data:

Transform your data (bronze to silver)

Extend the bronze-to-silver transformation pipeline to process data from your new source:

  • Customize the silver data model to accommodate the new data source.
  • Configure transformations specific to your new data source.

For detailed guidance on extending the transformation pipeline, see Extend data pipelines from one layer to another.

Enrich and analyze your data (silver to gold)

After your data is in the silver layer, you can create insights and segmentation:

  • Generate new enrichment insights.
  • Create the custom segmentation logic.

See Customize enrichment insights and Customize segmentation logic for detailed guidance.

Visualize your data

Make your new data available in reports:

  • Customize the semantic model to include tables and measures from your new data source.
  • Modify existing Power BI reports or create new ones to visualize the integrated data.
  • Create dashboards that combine insights from all your connected data sources.

See also

  • Last updated on