Data Factory - How to migrate a Snowflake Legacy connector to the New Snowflake connector

Joel Rueda González 5

What would be the correct way to modify a Linked Service type "Snowflake" to "SnowflakeV2" to not affect or change the Datasets related to the deprecated linked service (Snowflake Legacy).

Harishga 5,985 Reputation points Microsoft Vendor

2024-03-20T06:18:50.07+00:00

Hi @Joel Rueda González
Welcome to Microsoft Q&A platform and thanks for posting your question here.

To change from the Snowflake Legacy linked service to the new Snowflake connector SnowflakeV2, you need to make sure that your existing datasets are compatible with the new connector. Check if there are any custom expressions, transformations, or dependencies in your datasets that may not work with the new connector.

Then, create a new Linked Service of type "SnowflakeV2" and configure it to connect to your Snowflake account. For each dataset that uses the old Snowflake Legacy linked service, open the dataset definition and change the linked service reference to the new SnowflakeV2 linked service. Make sure that the dataset properties like schema, table, and columns are still correct, and save the changes.

After updating the datasets, test the new linked service by running sample data copy activities or data flows. Make sure that the data movement and transformations are the same as before.

Reference
https://learn.microsoft.com/en-us/azure/data-factory/connector-snowflake-legacy?tabs=data-factory

I hope this information helps you. Let me know if you have any further questions or concerns.
Vijay 11 Reputation points

2024-03-20T11:47:21.8+00:00
@Harishga Could you please highlight or give a brief on what will be changing between legacy and new Snowflake connector ? Also, would there be any timeframe till when the legacy connector will work ? thanks!
Harishga 5,985 Reputation points Microsoft Vendor

2024-03-20T13:18:26.1733333+00:00

Hi Vijay

The new SnowflakeV2 connector in Azure Data Factory has several key differences compared to the legacy Snowflake connector.

One of the most significant differences is that the new connector supports autoscaling, which allows it to automatically adjust its resources based on changes in traffic load. This means that the connector can scale out or in as needed to handle changes in data volume or processing requirements. The legacy connector does not support this feature.

Another important difference is that the new SnowflakeV2 connector can operate across multiple Availability Zones, making it more resilient to zone failures. This means that if one zone fails, the connector can automatically switch to another zone without any downtime. The legacy connector does not support this feature.

Finally, the new SnowflakeV2 connector supports the use of static IP addresses, which can be useful for security purposes. This allows you to use a static IP address for your connector instance, while the legacy connector does not support this feature.

Overall, the new SnowflakeV2 connector provides improved performance, scalability, and reliability compared to the legacy connector.

As for the timeframe for when the legacy connector will stop working, there is no specific date for when it will be deprecated or unsupported. It is recommended to migrate to the new SnowflakeV2 connector as soon as possible to take advantage of its improved features and performance.

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 5,985 Reputation points Microsoft Vendor

2024-03-21T05:05:01.24+00:00

@Joel Rueda González
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Joel Rueda Gonzalez 0 Reputation points

2024-03-21T07:32:04.1966667+00:00

Deleted answer as was duplicated with other personal account.
Joel Rueda González 5 Reputation points

2024-03-21T07:46:17.0466667+00:00

Hello @Harishga , thanks for your response and sorry for the delay on mine.

Actually, I can't even create a New Dataset with SnowflakeV2 connector as I'm getting the following error: Snowflake operation failed with error: '[Snowflake] arrow/ipc: could not read message schema: arrow/ipc: could not read message metadata: unexpected EOF'

I have previously created an Ingest Pipeline/Datasets through Copy Data tool (sinking to a Snowflake Legacy connector) and everything works fine, now that I perform the same steps with new connector is where I'm getting the error above.

Both Linked Services Test Connection is successful and both were configured equally (same account, credentials, etc), the only difference is the authentication type = Basic in new connector.

Integration Runtime was also upgraded to version 5.39.8830.4

When I try to edit a Legacy Dataset I can't see the New Linked Service, so I assume that it will not be compatible with V2 and if I'm not wrong I will have to create a new Dataset from scratch. If so, we expected that this Dataset could have been "reused" by changing the linked service and maybe some other minor adjustment.

Thanks in advance for your help, please point at me if I am performing anything incorrectly.

Regards!
Harishga 5,985 Reputation points Microsoft Vendor

2024-03-21T14:44:15.7166667+00:00
Hi @Joel Rueda González
Based on the error message you shared, it seems like there is an issue with the message schema or metadata when using the SnowflakeV2 connector. This error can occur due to various reasons, such as incorrect Snowflake account or credentials, network or firewall issues, or an issue with the previous Dataset or Linked Service.

To troubleshoot the issue, you can try the following steps:

Verify that the Snowflake account and credentials used in the Linked Service for the SnowflakeV2 connector are correct and have the necessary permissions to access the data.

Check if the Integration Runtime is running and has the correct version (5.39.8830.4) to support the SnowflakeV2 connector.

Verify that there are no network or firewall issues that may be preventing the connection to Snowflake. You can try connecting to Snowflake from a different network or machine to see if the issue persists.

When working with legacy datasets in Azure Data Factory, it's important to understand the compatibility and options available for transitioning to newer versions.

You are correct. Legacy datasets cannot be edited to use the new linked service, so you will need to create a new dataset from scratch with the new linked service. However, you can reuse the schema mapping from the legacy dataset to make the process easier.

Legacy datasets may not automatically support new features or linked services introduced in v2. Thats why you're unable to see the new linked service when editing a legacy dataset, it's likely due to compatibility limitations.

I hope this information helps you. Let me know if you have any further questions or concerns.
Joel Rueda González 5 Reputation points

2024-03-25T08:34:08.6966667+00:00

Hello @Harishga and thanks again for your response.

I did some more research and discovered that perhaps there could be an issue with SnowflakeV2 connector when trying to retrieve the tables from Snowflake as this is happening in both Copy Data Tool wizard and New Dataset creation.

ADF Copy Data tool

When specifying a SnowflakeV2 connection we are getting the mentioned error but when directly querying to the same connection we are not getting any issue to retrieve some data.

Checking the above in Snowflake query history, we confirm that both executions were successful.

ADF Create New Dataset

As loading tables is failing, we are forced to select Enter manually option and specify a existing schema.table from database.

Once a schema.table is entered we can parameterize the dataset, but this does not exclude the fact that the tables preloading is not working properly.

Finally, I confirm you that we have the most recent version for the Integration Runtime.

From our side we consider that we don't have any issues with either the credentials or the network/firewall, otherwise we would not have had a satisfactory response with the Query option. If you think something else needs to be checked in our data factory please help us with your feedback.

Thanks in advance for your support investigating this possible issue with the V2 connector as Copy Data tool have helped us a lot when creating ingests of several tables from scratch.

Regards!
Subash Sagi 0 Reputation points

2024-03-29T19:09:59.35+00:00

Hi @Harishga , thanks for sharing advantages of using the Snowflake connector V2, Could you please share any Microsoft documentation listing all the advantages over the legacy connector.
Harishga 5,985 Reputation points Microsoft Vendor

2024-04-01T12:43:56.4233333+00:00

Hi Subash Sagi
Unfortunately, there is no Microsoft official documentation that directly compares new features or connectors with their legacy versions. The documentation for each individual feature or connector can provide information about the differences between the new and legacy versions, as well as any advantages or improvements.

Share via

Data Factory - How to migrate a Snowflake Legacy connector to the New Snowflake connector