Pipeline runs issue

Question

Pipeline runs issue

Stephen 26

There are two linked service connection failures in Azure Data Factory. Since we are currently using Version 1 and planning to upgrade to Version 2, is there anything we should be cautious about before making the change? Do you have any guidelines we can follow? Thank you!

J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-03T00:34:30.79+00:00
Hi @Stephen
Please help me with below questions So that I can provide the best solution?

What types of data sources and sinks are used in your current linked services (e.g., Azure SQL, Oracle, Blob Storage)?

What type of authentication do your linked services use (e.g., username/password, certificate, managed identity)?

Are you planning a full redesign in ADF V2 or a direct migration of existing pipelines?

Do you require Git integration or CI/CD for managing ADF assets?

Are you currently using or planning to use a self-hosted Integration Runtime (IR)?

What is your expected timeline and are there any blockers (e.g., network/firewall constraints, lack of access)?

Check the full error messages for the two failing linked services in the ADF portal or pipeline run logs.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-03T00:47:33.4466667+00:00

@Stephen

Upgrading from Version 1 to Version 2 linked services in Azure Data Factory (ADF) to address connection failures requires careful planning to ensure a smooth transition and avoid pipeline disruptions. Below are key considerations, potential issues to be cautious about, and guidelines for the upgrade process

Asses Current Environment

List all existing pipelines, datasets, and linked services in ADF V1. Identify activities unsupported in V1 but potentially simplified in V2.

Linked service

Pipelines and linked services must be recreated or reconfigured manually in V2. Redesign logic using IF Condition,Foreach,Until etc , available in V2.
Review the failed linked services to determine if they require credential updates, firewall changes, or support for newer features in ADF V2.

Authentication

Validate that secure connections using passwords or keys are handles via Azure Keyvault in V2. Confirm firewall, Vnet, or private endpoints are compatible with V2 integration runtime.

Testing&staging

create a dev/staging ADF V2 instance and test all migrated pipelines.validate pipeline runs, triggers, data movement and transformation.

Monitor and Alert setup

Reconfigure monitoring, alerts, and logging as V2 uses a new metrics system. We can also use Azure monitor or Log Analytics.
Review the and assign RABC roles specific to ADF V2.

If you provide details about the data sources or error messages for the failing linked services, I can offer more specific advice.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-05T02:04:24.2466667+00:00

@Stephen We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Stephen 26 Reputation points

2025-05-05T16:29:29.2366667+00:00

@J N S S Kasyap and @phemanth Thank you all so much. Since this part of the services isn't something I typically handle, I'm currently just gathering information. I truly appreciate your input and will follow up with more details as soon as I have them.

Accepted answer

1 additional answer

Your answer

J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-03T00:34:30.79+00:00

Hi @Stephen
Please help me with below questions So that I can provide the best solution?

What types of data sources and sinks are used in your current linked services (e.g., Azure SQL, Oracle, Blob Storage)?

What type of authentication do your linked services use (e.g., username/password, certificate, managed identity)?

Are you planning a full redesign in ADF V2 or a direct migration of existing pipelines?

Do you require Git integration or CI/CD for managing ADF assets?

Are you currently using or planning to use a self-hosted Integration Runtime (IR)?

What is your expected timeline and are there any blockers (e.g., network/firewall constraints, lack of access)?

Check the full error messages for the two failing linked services in the ADF portal or pipeline run logs.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-03T00:47:33.4466667+00:00

@Stephen

Upgrading from Version 1 to Version 2 linked services in Azure Data Factory (ADF) to address connection failures requires careful planning to ensure a smooth transition and avoid pipeline disruptions. Below are key considerations, potential issues to be cautious about, and guidelines for the upgrade process

Asses Current Environment

List all existing pipelines, datasets, and linked services in ADF V1. Identify activities unsupported in V1 but potentially simplified in V2.

Linked service

Pipelines and linked services must be recreated or reconfigured manually in V2. Redesign logic using IF Condition,Foreach,Until etc , available in V2.
Review the failed linked services to determine if they require credential updates, firewall changes, or support for newer features in ADF V2.

Authentication

Validate that secure connections using passwords or keys are handles via Azure Keyvault in V2. Confirm firewall, Vnet, or private endpoints are compatible with V2 integration runtime.

Testing&staging

create a dev/staging ADF V2 instance and test all migrated pipelines.validate pipeline runs, triggers, data movement and transformation.

Monitor and Alert setup

Reconfigure monitoring, alerts, and logging as V2 uses a new metrics system. We can also use Azure monitor or Log Analytics.
Review the and assign RABC roles specific to ADF V2.

If you provide details about the data sources or error messages for the failing linked services, I can offer more specific advice.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2025-05-05T02:04:24.2466667+00:00

@Stephen We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Stephen 26 Reputation points

2025-05-05T16:29:29.2366667+00:00

@J N S S Kasyap and @phemanth Thank you all so much. Since this part of the services isn't something I typically handle, I'm currently just gathering information. I truly appreciate your input and will follow up with more details as soon as I have them.

Answer 1

Hi Stephen ,

Since you are planning to upgrade your connector in ADF from version 1 to version2 , you can opt for using Connector upgrade advisor in ADF.

The Connector upgrade advisor is a comprehensive tool that helps upgrade pipelines using outdated V1 linked services and datasets to their latest available versions . Before using the advisor, save or discard all pending changes. The advisor scans your data factory and identifies all existing pipelines using outdated connectors along with the number of dependencies on outdated versions .

Screenshot of the connector upgrade advisor tool page.

Here is how to Connector upgrade advisor :

Navigate to the Manage pane in your data factory and select Connector upgrade advisor

Select Pipelines: Choose the pipelines you want to upgrade and click Upgrade 1. If a selected pipeline references an outdated dataset shared with other unselected pipelines, you can choose to upgrade all pipelines using the same dataset simultaneously or create new datasets for selected pipelines
Map Linked Services: Map outdated linked services to new linked services by selecting an equivalent V2 linked service from the dropdown menu or creating a new one that supports the V2 format
Update Queries: If your selected pipelines use Salesforce, Salesforce Service Cloud, or ServiceNow connectors in the Copy activity with a query, manual adjustments may be required due to query format differences between V1 and V2
Review Changes: Review and finalize changes using the Review page, then save or publish all modifications Known Limitations: Currently, support for the Snowflake data source is unavailable

Reference: Connector upgrade advisor in Azure Data Factory and Azure Synapse Analytics

Upgrade plan for Azure Data Factory connectors

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Stephen 26 Reputation points

2025-05-05T16:30:34.9166667+00:00

@AnnuKumari-MSFT I will give a try on this. Thanks!
Smaran Thoomu 24,110 Reputation points Microsoft External Staff Moderator

2025-05-06T09:07:49.89+00:00

@Stephen We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
J N S S Kasyap 3,625 Reputation points Microsoft External Staff Moderator

2025-05-07T09:06:07.8833333+00:00

@Stephen We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 2

Stephen 26

Thank you for all your support. I’m not fully clear on the scope of the project at this stage, so I will delegate the tasks to the appropriate team to handle moving forward. Thanks!

Share via

Pipeline runs issue

1 additional answer

Your answer