Are linked service parameters disappearing now?

Question

Are linked service parameters disappearing now?

Brent Leslie 396

It feels like something has changed in data factory linked service parameters recently. I have setup my data factory as follows:

I have created a linked service pointing to on-prem SQL boxes
I have added parameters to this linked service, using the "+ New" parameter box for Server/DB/User
I have then referenced these linked service parameters in the linked service Server/DB name/SQL Auth boxes:

This all works fine. This forms part of a release pipeline. I deployed this to UAT and noticed UAT no longer has most of these parameters specified, however it does have the Key Vault parameter:

The parameter definitions come through to UAT, but not the link to these parameters in the linked services:

I can also confirm this, when I Export the templates from the user interface in Data Factory, I check the ArmTemplate_0.json file (in the Linked Templates folder of the exported file) and the arm_template.json file from the root, to find where the Linked Service parameters should be specified. All I see is this in both files:

I would also expect to see additional "typeProperties" for the other parameters specified.

This seems to confirm that when specifying Linked Service parameters, not all values are being saved in the arm templates files. Only Azure Key Vault parameters are being saved? I just tried doing the same thing with a different data factory and get the same result. I thought there was something wrong with my CI/CD process, however it looks like the problem exists even without going through Azure DevOps release processes.

Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-04T01:25:21.483+00:00

@Brent Leslie Thanks for using Microsoft Q&A.

Sorry for your trouble. I am looking this issue and get back to you with findings.
Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-04T19:33:59.453+00:00

@Brent Leslie I am able to reproduce this in my environment (without CI/CD) and checking internally with the products team on this. I will update you here as soon as I hear back from them.
Brent Leslie 396 Reputation points

2020-12-06T21:10:33.247+00:00

Cool thanks @SaurabhSharma0303 - it's broken a few of our ETL processes at the moment. So currently we have a few failing processes which previously used to work. I think I can work around them by manually publishing the parameters, but I will have to do this every evening due to us continually updating our data factories with nightly builds.
Brent Leslie 396 Reputation points

2020-12-09T19:32:23.097+00:00

@Saurabh Sharma - any updates on this?
Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-09T23:26:08.847+00:00

@Brent Leslie sorry for the delay on this. I am still waiting for products team response this. I will update you as soon as I get a response.

Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-04T01:25:21.483+00:00

@Brent Leslie Thanks for using Microsoft Q&A.

Sorry for your trouble. I am looking this issue and get back to you with findings.
Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-04T19:33:59.453+00:00

@Brent Leslie I am able to reproduce this in my environment (without CI/CD) and checking internally with the products team on this. I will update you here as soon as I hear back from them.
Brent Leslie 396 Reputation points

2020-12-06T21:10:33.247+00:00

Cool thanks @SaurabhSharma0303 - it's broken a few of our ETL processes at the moment. So currently we have a few failing processes which previously used to work. I think I can work around them by manually publishing the parameters, but I will have to do this every evening due to us continually updating our data factories with nightly builds.
Brent Leslie 396 Reputation points

2020-12-09T19:32:23.097+00:00

@Saurabh Sharma - any updates on this?
Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-09T23:26:08.847+00:00

@Brent Leslie sorry for the delay on this. I am still waiting for products team response this. I will update you as soon as I get a response.

Answer 1

Saurabh Sharma 17,321 Microsoft Employee

@Brent Leslie Here is the response from the products team -
Connection string is treated as a sensitive field even though the password is specified with a akv. In this case, the connection string value is stored in an arm template parameter named "onprem_SQL_Server_linked_service_use_parameters_connectionString" and it should look like below

So, you need to either manually edit the arm template to give a default value to the "onprem_SQL_Server_linked_service_use_parameters_connectionString" or overwrite the parameter value in the deployment pipeline like

We tried the second method and the linked service was successfully deployed like shown below-

Please let me know if you have any other questions.

Brent Leslie 396 Reputation points

2020-12-16T03:30:50.773+00:00

Thanks SaurabhSharma-msft - I have implemented your suggested solution which looks to be working. I did need to put in a "TrustServerCertificate=True" into the connection string as well (not a problem as the host is in Azure on a VM as well, so nothing across the public network). Obviously this would be a problem if we were true "on prem".

There is something odd going on with DevOps releases and Linked Services which I am struggling to track down (sometimes they are overwritten on release, sometimes they aren't). If I am able to get to the source of it, I will get back to you.
Saurabh Sharma 17,321 Reputation points Microsoft Employee

2020-12-16T18:48:34.447+00:00

@Brent Leslie Great to hear the solution works for you. Sure, please let me know over here or open a new Q&A thread for any other issues you see.

----------

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.
Aayush Ku Soni 1 Reputation point

2022-04-08T10:07:03.3+00:00

Hello Team, I did tried 2nd method. But then database name is not coming as expected. It's getting created in "Additional connection properties". Can you please help me out for this.
Fang Liu (ADMS) 1 Reputation point Microsoft Employee

2022-04-18T14:26:33.523+00:00

Database should be "Initial Catalog". What's your connection string?
Omkar Ghaisas 1 Reputation point

2022-11-21T15:24:45.973+00:00

@Saurabh Sharma @Brent Leslie - I have the same requirement where the ADO deploy pipeline is trying to deploy the ADF resource and within the ARM template, I need to pass on the Database (Initial Catalog) as @{linkedService().DB_name}, but even after using the 1st way of setting the defaultValue directly on the main ARM template file, I keep getting a deployment failed with the ARM template.

It keeps pointing out - Missing parameter definition DB_name (BadRequest).

Can someone, please share the sample of this ARM template and ARM template parameter file? Am I doing something wrong here? As this solution seems to be working for deploying the ARM template with linked service containing parameter.

Any help is much appreciated.
Shahid Awan 0 Reputation points

2023-02-25T15:20:26.5833333+00:00

The above png files 46968-image.png, 46969-image.png and 47071-image.png pictures are not showing up, can anyone please upload them again? thanks
Fang Liu (ADMS) 1 Reputation point Microsoft Employee

2023-03-02T12:25:05.7766667+00:00

I believe this issue has been fixed. Are you still hitting this? What's your linked service payload? (Share the JSON code)
Fang Liu (ADMS) 1 Reputation point Microsoft Employee

2023-03-02T12:26:49.2133333+00:00

I believe this has been fixed. Are you still hitting this?
Naveen Kuppili 41 Reputation points

2023-03-22T00:14:35.49+00:00

@Fang Liu (ADMS) I know this old thread, I have same issue as Brent with the parameters disappearing @Shahid Awan png not showing up anymore. Can someone upload them please?

Are linked service parameters disappearing now?

0 additional answers

Activity