Azure Data Factory, Dataset using parameters

Question

What I am trying to do is create a dataset/pipeline which set the parameters dynamically from the name of the Data Factory.

So I have a 3 data Factories, all named exactly the same except for the instance designation.

Apparently I can get that information from using the command, @pipeline().DataFactory. (I still haven't found a method that tells me what is actually being held however, that is yet another question)

From this I should be able get the instance, for example, the names of the Data factories are as follows:

dataFactory_d
dataFactory_t
dataFactory_p

If I can then use substring(@pipeline().DataFactory,13,1) this should give me either 'd', 't' or 'p' as the Instance. Depending on whether the array is 0 based or not, again I have no idea from reading the guides. This would be saved as the parameter, dbInstance.

So once I have this information, I would then have something that I can use to dynamically determine which structure to use for the connection.

I have set up these so that they are using the Linked service, and I can manually fill in the values and the connections work fine. for example:-

DomainName = domainDev
UserName = userDev
SecretName = sectretDev

What I am now trying to achieve is to set up the values of the Parameters: if(equals(@{dataset().dbInstance}),'p'),'db_prod',equals(@{dataset().dbInstance}),'t'),'db_test','db_dev')

If dataset().dbInstance = 'p' then dataset().DomainName = 'db_prod'
If dataset().dbInstance = 't' then dataset().DomainName = 'db_test'
otherwise dataset().DomainName = 'db_dev'

I would then be able to set the linked service parameter == dynamic parameter.
@{linkedService().DomainName} = @{dataset().DomainName}

This method doesn't seem to work. I am getting this error:

Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access

I'm guessing it has to be something to do with how I am trying to pass the parameters around, but I cannot figure out how to get around this, as I haven't managed to find a way to see what the values are of these parameters. When the values are hardcoded in the linked Service screens, then the method works.

Firstly, does this sound feasible? and has anybody else done this and how did they get around this issue.

Many thanks

Answer 1

It will be complicated this way I feel.

Another better option is to create a dev data factory (associated to GIT repository). This ADF linked services connects to Dev resources like your DEV SQL DB etc..

Using Azure DevOps release, you can deploy to PPE and PROD ADF where release will replace all the connections (like SQL Server name, DB name to point to PPE, PROD resources)

Reference - https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#cicd-lifecycle

Answer 2

Hi, the answer by @Vaibhav Chaudhari would definitely solve the problem. I have also tried creating variables, but this method doesn't seem to work as they it seems that I can't uses these with the linked service for db connections.