Azure Data Factory Copy Data "Failing" after upgrading Snowflake Connector

Question

I have a couple ADF pipelines that do simple Copy Data operations, essentially truncating and loading data from SQL Server to Snowflake. These have been running for a couple years with little to no issues. Since upgrading the Snowflake Connector (per ADF recommendations) the other day, these have been "failing" with this error code: ErrorCode=GenericAdoNetReadError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to execute the query command during read operation.,Source=Microsoft.DataTransfer.Connectors.GenericAdoNet,''Type=Apache.Arrow.Adbc.AdbcException,Message=[Snowflake] arrow/ipc: could not read message schema: arrow/ipc: could not read message metadata: unexpected EOF,Source=Apache.Arrow.Adbc,'

I put "failing" in quotes because it looks like the jobs actually work. All the expected rows of data get written to Snowflake, but because it fails, downstream steps haven't been running.

Thanks for any help or suggestions!

Answer 1

Hello @Bobby Neelon

The error message you provided suggests that there is an issue with the query command during the read operation.

One possible reason for this error could be that the upgraded Snowflake connector is not compatible with the version of Snowflake you are using.

You can check the compatibility matrix for the Snowflake connector in Azure Data Factory to ensure that you are using a compatible version.

Another possible reason could be that there is an issue with the metadata of the message schema. This could be caused by a corrupt or incomplete message schema. You can try to regenerate the message schema to see if that resolves the issue.

If neither of these solutions work, you can try to troubleshoot the issue further by enabling diagnostic logging for the Snowflake connector in Azure Data Factory. This will provide more detailed information about the error and help you identify the root cause.

I hope this helps