Data Flow in Azure Data Factory

Lakshmi Suryadevara 21 Reputation points Microsoft Employee
2024-03-21T17:11:00.08+00:00

The copy activity successfully facilitates data transfer bidirectionally between an on-premises SQL Server behind a private endpoint. However, the data flow process encounters issues. What could be the underlying cause for this discrepancy?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,490 questions
{count} votes

Accepted answer
  1. Amira Bedhiafi 15,041 Reputation points
    2024-03-21T22:25:30.37+00:00

    When it comes to issues related to discrepancy, I can say that in the 1st place comes the environment configuration : issues like the config of runtime (self-hosted or Azure) used by the data flow has proper access to the on-premises SQL Server (involving also firewall and NSG).

    Without forgetting also, the insufficient resources or misconfigurations in the IR settings.

    Also the transformation logic can add a layer of complexity. Within data flows might encounter issues if there are data type mismatches, complex expressions that fail, or unsupported operations for the source or destination.

    Security and persmissions ? Yes I won't deny that also.

    Don't forget also the resources limitations since ADF has quotas and limits for concurrent activities, debug sessions, and data flow executions. If the data flows are large or complex, they might hit these limits, causing failures or performance issues.

    0 comments No comments

0 additional answers

Sort by: Most helpful