ADF Dataflow and Linked Service

Question

I created a linked service to connect snowflake. After providing the connection information, I check the 'test connection' directly from linked service and it shows connection successful.
Now when I use the same linked service inside adf dataflow , the connection fails .
I am creating all these thinks inside my working branch in ADF and nothing is deployed in Dev branch.
Please help me to identify the cause, why dataflow is not able to connect to snowflake and connection failed from within dataflow but connection succeeded in linked service.
Is it necessary to use a linked service which is already deployed.

Answer 1

Hi @Amit Trivedi

Thanks for contacting to Microsoft QA. It looks like your Azure Data Factory (ADF) Dataflow is having trouble connecting to Snowflake, even though the linked service test passed successfully. Here are some possible reasons why this might be happening:
Check Integration Runtime (IR)

Make sure the Integration Runtime (IR) you’ve set in your linked service matches the one used in your Dataflow activity. Remember, Dataflow doesn’t rely on the IR from the linked service—it uses the IR defined at the activity level. Double-check this setting, especially when running a debug session.

Debug Mode Settings

If you’re running the Dataflow in debug mode, confirm that you’ve selected the correct IR as per the guidelines. For detailed steps, refer to the Azure documentation on Mapping Data Flow Debug Mode. Mapping data flow Debug Mode - Azure Data Factory & Azure Synapse | Microsoft Learn

Connectivity Considerations

Dataflows can have limitations with certain data stores or network setups, like firewalls or private endpoints. Verify that your Snowflake environment isn’t affected by these restrictions. For instance, Dataflows work with Azure integration runtimes and managed VNets but don’t support on-premises stores or those behind strict firewall rules.

Publish Your Changes

Since you mentioned working in your branch without deploying to the Dev branch, ensure all updates to your linked service and Dataflow are published. The linked service must be properly configured and deployed for Dataflow to access it.

Parameter Setup

If you’re using parameters in your linked service, make sure they’re configured correctly and that the right values are being passed into your Dataflow.

Ref: -

Mapping Data Flows in Azure Data Factory

Common Issues and Solutions

Data Access Strategies through Azure Data Factory

Answer 2

The issue you're experiencing with the Azure Data Factory (ADF) data flow failing to connect to Snowflake, despite the linked service connection being successful, could be due to several reasons:

1. Integration Runtime: When you test the connection in the linked service, it may be using a different integration runtime than the one used in the data flow. Ensure that the integration runtime configured for the data flow is the same as that used for the linked service test.
2. Network Configuration: The data flow might be running in a different network environment that does not have access to Snowflake. Check your network settings and firewall rules to ensure that the IP addresses used by ADF data flows are allowed to access Snowflake.
3. Permissions: The user/role configured in the linked service may not have the necessary permissions to perform operations in the data flow context. Ensure that the role has the required privileges to access the Snowflake resources used in the data flow.
4. Temporary Staging: When using data flows, ADF creates a temporary stage in Snowflake to pull data. If the user/role does not have permissions to create or access this temporary stage, it could lead to connection failures.
5. Deployment State: While it is not strictly necessary to use a linked service that is already deployed, if there are any configuration changes made in the working branch that have not been published or deployed, they might not reflect in the data flow execution.

To troubleshoot, you can check the following:

• Verify the integration runtime settings for both the linked service and data flow.
• Review the network access settings in Snowflake to ensure the ADF IP ranges are allowed.
• Confirm that the user/role has the necessary permissions in Snowflake.
• Ensure that any changes made in the working branch are published and deployed correctly.

If the issue persists, consider reviewing the error messages in the ADF monitoring section for more specific details on the failure.