Azure Synapse Analytics- Overriding SelfHost IR Parameter

Question

Azure Synapse Analytics- Overriding SelfHost IR Parameter

Sarthak 5

Hi Team,

Im facing issue with Azure Synapse Pipeline deployment through Azure DevOps CI/CD deployment pipeline to higher environment with Self Host Integration Run time. Here is the scenario:

Data is getting copied to Azure Synapse storage from On-Prem Oracle System and one SFT server. This is done using a Self-Hosted IR
The file: template-parameters-definition.json has been updated accordingly :

 "Microsoft.Synapse/workspaces/linkedServices": {
       
        "Oracle": {
            "properties": {
                "typeProperties": {
                    "connectionString": "=",
                    "password": {
                        "type": "=",
                        "store": {
                            "referenceName": "="
                        },
                        "secretName": "="
                    }
                },
                "connectVia": {
                    "referenceName": "="
                }
            }
        },
        "Sftp": {
            "properties": {
                "typeProperties": {
                    "host": "=",
                    "userName": "=",
                    "password": {
                            "type": "=",
                            "store": {
                                "referenceName": "="
                            },
                            "secretName": "=",
                            "secretVersion": "="
                        }
                },
                "connectVia": {
                    "referenceName": "="
                }
            }
        }

as seen above have included the property: "connectVia"

Now in the Work_publish folder that contains these two files: TemplateForWorkspace.json & TemplateParametersForWorkspace.json has the entries accordingly after the code gets published from the git main branch

-- This is for TemplateParametersForWorkspace.json

"LS_Oracle_IFS_properties_connectVia_referenceName": { 			"value": "Integration-runtime-selfhost-eusfs02" 		}, "LS_Sftp_eus_01_properties_connectVia_referenceName": { 			"value": "Integration-runtime-selfhost-eusfs02" 		}

-- This is for TemplateForWorkspace.json

   {
               "name": "[concat(parameters('workspaceName'), '/LS_Oracle_IFS')]",
               "type": "Microsoft.Synapse/workspaces/linkedServices",
               "apiVersion": "2019-06-01-preview",
               "properties": {
                   "annotations": [],
                   "type": "Oracle",
                   "typeProperties": {
                       "connectionString": {
                           "type": "SecureString",
                           "value": "[parameters('LS_Oracle_IFS_properties_typeProperties_connectionString')]"
                       },
                       "password": {
                           "type": "[parameters('LS_Oracle_IFS_properties_typeProperties_password_type')]",
                           "store": {
                               "referenceName": "[parameters('LS_Oracle_IFS_properties_typeProperties_password_store_referenceName')]",
                               "type": "LinkedServiceReference"
                           },
                           "secretName": "[parameters('LS_Oracle_IFS_properties_typeProperties_password_secretName')]"
                       }
                   },
                   "connectVia": {
                       "referenceName": "[parameters('LS_Oracle_IFS_properties_connectVia_referenceName')]",
                       "type": "IntegrationRuntimeReference"
                   }
               },
               "dependsOn": [
                   "[concat(variables('workspaceId'), '/integrationRuntimes/', parameters('LS_Oracle_IFS_properties_connectVia_referenceName'))]",
                   "[concat(variables('workspaceId'), '/linkedServices/', parameters('LS_Oracle_IFS_properties_typeProperties_password_store_referenceName'))]"
               ]
           }

As seen above the dependsOn and connectVia property has been updated in the file: TemplateForWorkspace.json

Now with this setup while going for deployment to our UAT environment im getting error. Logs attached here

summary:
"Encountered with exception:Error: Failed to parse package: Error: Could not figure out full dependancy model."

2023-12-06T12:44:54.4805277Z Could not figure out full dependancy model for these artifacts. Check template for correctness.
2023-12-06T12:44:54.4805753Z -----------------------------------------------------------------------------------------------
2023-12-06T12:44:54.4806168Z Name: PL_Copy_Tables, Type: Microsoft.Synapse/workspaces/pipelines
2023-12-06T12:44:54.4806611Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/PL_File_Transfer_bulk
2023-12-06T12:44:54.4807105Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/datasets/OracleTable1
2023-12-06T12:44:54.4807529Z Name: PL_DependentTable_Load, Type: Microsoft.Synapse/workspaces/pipelines
2023-12-06T12:44:54.4808001Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/PL_Src_To_Landing
2023-12-06T12:44:54.4808450Z Name: PL_File_Transfer_bulk, Type: Microsoft.Synapse/workspaces/pipelines
2023-12-06T12:44:54.4808880Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/datasets/Binary1
2023-12-06T12:44:54.4809537Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/datasets/DS_Landing_Source
2023-12-06T12:44:54.4810039Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/datasets/DS_cpy_to_processed_target
2023-12-06T12:44:54.4810483Z Name: PL_Src_To_Landing, Type: Microsoft.Synapse/workspaces/pipelines
2023-12-06T12:44:54.4811114Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/PL_Copy_Tables
2023-12-06T12:44:54.4811537Z Name: pl_master_data_Load, Type: Microsoft.Synapse/workspaces/pipelines
2023-12-06T12:44:54.4813696Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/PL_Src_To_Landing
2023-12-06T12:44:54.4814151Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/PL_DependentTable_Load
2023-12-06T12:44:54.4814619Z Name: Binary1, Type: Microsoft.Synapse/workspaces/datasets
2023-12-06T12:44:54.4815091Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/linkedServices/LS_Sftp_eus_01
2023-12-06T12:44:54.4815540Z Name: DS_Landing_Source, Type: Microsoft.Synapse/workspaces/datasets
2023-12-06T12:44:54.4815999Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/linkedServices/LS_Sftp_eus_01
2023-12-06T12:44:54.4816436Z Name: DS_cpy_to_processed_target, Type: Microsoft.Synapse/workspaces/datasets
2023-12-06T12:44:54.4816891Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/linkedServices/LS_Sftp_eus_01
2023-12-06T12:44:54.4817346Z Name: OracleTable1, Type: Microsoft.Synapse/workspaces/datasets
2023-12-06T12:44:54.4817772Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/linkedServices/LS_Oracle_IFS
2023-12-06T12:44:54.4818217Z Name: LS_Oracle_IFS, Type: Microsoft.Synapse/workspaces/linkedServices
2023-12-06T12:44:54.4818675Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/integrationRuntimes/On-prem-Selfhosted
2023-12-06T12:44:54.4819319Z Name: LS_Sftp_eus_01, Type: Microsoft.Synapse/workspaces/linkedServices
2023-12-06T12:44:54.4819766Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/integrationRuntimes/On-prem-Selfhosted
2023-12-06T12:44:54.4820217Z Name: Trg_Ceridian, Type: Microsoft.Synapse/workspaces/triggers
2023-12-06T12:44:54.4820599Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/pl_master_data_Load
2023-12-06T12:44:54.4820939Z Name: Trg_Connect, Type: Microsoft.Synapse/workspaces/triggers
2023-12-06T12:44:54.4821276Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/pl_master_data_Load
2023-12-06T12:44:54.4821623Z Name: Trg_IFS_Full_Load, Type: Microsoft.Synapse/workspaces/triggers
2023-12-06T12:44:54.4822024Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/pl_master_data_Load
2023-12-06T12:44:54.4822439Z Name: Trg_IFS_Incr_Load, Type: Microsoft.Synapse/workspaces/triggers
2023-12-06T12:44:54.4822791Z     Dependency Not found: Microsoft.Synapse/workspaces/ws-da-edw-finhr-qa-01/pipelines/pl_master_data_Load
2023-12-06T12:44:54.4823115Z -----------------------------------------------------------------------------------------------
2023-12-06T12:44:54.4834504Z deploy operation failed
2023-12-06T12:44:54.4835155Z An error occurred during execution: Error: Failed to parse package: Error: Could not figure out full dependancy model. Some dependancies may not exist in template.
2023-12-06T12:44:54.4836486Z ##[debug]task result: Failed
2023-12-06T12:44:54.4864999Z ##[error]Encountered with exception:Error: Failed to parse package: Error: Could not figure out full dependancy model. Some dependancies may not exist in template.
2023-12-06T12:44:54.4871598Z ##[debug]Processed: ##vso[task.issue type=error;]Encountered with exception:Error: Failed to parse package: Error: Could not figure out full dependancy model. Some dependancies may not exist in template.
2023-12-06T12:44:54.4878625Z ##[debug]Processed: ##vso[task.complete result=Failed;]Encountered with exception:Error: Failed to parse package: Error: Could not figure out full dependancy model. Some dependancies may not exist in template.
2023-12-06T12:44:54.5030161Z ##[section]Finishing: Synpase deployment task for workspace: ws-da-edw-finhr-qa-01

Please let me know what might be the issue and how do i fix it. Please advice.

Note: If i'm remove this self-host integration run time parametrization from template-parameters-definition.json then the deployment goes fine, but i had to manually set or reconfigure the SHIR refrenece in the linked service connection it in higher environemnt (UAT) and then publish the code again.

Sarthak 5 Reputation points

2023-12-13T05:53:03.1733333+00:00

Any help/suggestion for this issue.
AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator

2023-12-14T06:29:33.13+00:00

Hi Sarthak ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to deploy synapse workspace ARM template from Dev environment to UAT environment, however during this process you are experiencing an error "Could not figure out full dependency model for these artifacts" related to Integration runtime. Please let me know if that is not the case.

It seems that all the dependencies related to the IR properties are not exposed by default in the templateParametersForWorkspace.json and templateForWorkspace.json files.

Could you please try creating a custom parameter template which would include all the dependencies related to the IR like location computeType, coreCount, TimeToLive, etc.

For more details on how to use CustomParameter template , kindly checkout the following resources:

Synapse CI/CD: Using a custom parameter template

Create custom parameters in the workspace template

Hope it helps. Kindly let us know how it goes. Thankyou
Sarthak 5 Reputation points

2023-12-14T14:47:52.7666667+00:00
Hi Annu,

Thanks! for your response. I need your help in adding below mentioned properties to custom template parameter definition

"Could you please try creating a custom parameter template which would include all the dependencies related to the IR like location computeType, coreCount, TimeToLive, etc."

Because in the code definition of Self-Hosted IR i'm getting only these settings:

{ "name": "Integration-runtime-selfhost-eusfs02", "properties": { "type": "SelfHosted" } }

And here is the setup:

How do i include properties : location computeType, coreCount, TimeToLive to custom template parameter definition for integrationruntimes tag.

If you can please share some reference or links to do this custom setup that will help in my synapse pipeline deployment for Integrationruntimes to higher environment.
Sarthak 5 Reputation points

2023-12-18T08:32:24.04+00:00

Any update on my query AnnuKumari-MSFT
AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator

2023-12-19T09:10:22.45+00:00

Hi Sarthak ,

Kindly checkout the above video which has the end to end guidance on using custom parameter template. Sample json including the dependencies can be seen in the below image:

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Sarthak 5

Hi,

I tried to use custom template-parameters-definition.json file for making entry Integration times

  "Microsoft.Synapse/workspaces/integrationRuntimes": {
        "properties": {
            "typeProperties": {
                "*": "="
            }
        }
    }

But in the template-parameter-defintion.json file here is the output i'm getting after i publish the code from main branch

   "AutoResolveIntegrationRuntime_properties_typeProperties_computeProperties": {
            "value": {
                "location": "AutoResolve",
                "dataFlowProperties": {
                    "computeType": "General",
                    "coreCount": 8,
                    "timeToLive": 0
                }
            }
        }

There is no entry / reference for the self-hosted integration runtime

Integration-runtime-selfhost-eusfs02

.

Please let me know if i'm missing something here...Please advice

Sarthak 5 Reputation points

2023-12-26T07:24:48.0766667+00:00

@AnnuKumari-MSFT , Appreciate if you could please provide any guidance for this deployment issue..

AnnuKumari-MSFT 34,561 Microsoft Employee Moderator

Sarthak ,

Kindly check ARM template resource definition ,

{
  "type": "Microsoft.DataFactory/factories/integrationRuntimes",
  "apiVersion": "2018-06-01",
  "name": "string",
  "properties": {
    "description": "string",
    "type": "string"
    // For remaining properties, see IntegrationRuntime objects
  }
}

For properties of IntegrationRuntime objects, Set the type property to specify the type of object.

For SelfHosted, use:

"type": "SelfHosted",
  "typeProperties": {
    "linkedInfo": {
      "authorizationType": "string"
      // For remaining properties, see LinkedIntegrationRuntimeType objects
    }
  }

Hope it helps. Thankyou

Sarthak 5

Here is the updated section for Integration runtime in file: template-parameters-definition.json ( as per the documentation and steps defined for SHIR)

"Microsoft.Synapse/workspaces/integrationRuntimes": {             
"name": "=",             
"properties": {             
	"type": "SelfHosted", 	    
		"typeProperties": {                
				"linkedInfo": {                  
					 "authorizationType": "key"                
								}             
						  }         
				}     
}

And after i publish from the main branch , here is the output captured in file: TemplateParametersForWorkspace.json

"AutoResolveIntegrationRuntime_name": {
            "value": "[concat(parameters('workspaceName'), '/AutoResolveIntegrationRuntime')]"
        },
 "Integration-runtime-selfhost-eusfs02_name": {
            "value": "[concat(parameters('workspaceName'), '/Integration-runtime-selfhost-eusfs02')]"
        }

This wont help in the deployment through DevOps pipeline, i need to set the correct Integration runtime reference in the Linked service connection again at higher environment and republish

Any other suggestion or am i missing something here..

Sarthak 5 Reputation points

2024-01-08T13:32:00.2133333+00:00

Hi @AnnuKumari-MSFT please provide your input / suggestions. I tried to implement those steps (mentioned here in this link :https://learn.microsoft.com/en-us/azure/templates/microsoft.datafactory/factories/integrationruntimes?pivots=deployment-language-arm-template#resource-format-1. but it didn't work.
Wasim 20 Reputation points

2024-02-01T23:54:34.8166667+00:00

HI @sarthak , were you able to resolve the issue? I am faced with same issue, wonder how you resolved if you have already.
Sarthak 5 Reputation points

2024-02-05T16:42:25.8033333+00:00

Issue is still there, I had to manually update the Linked service connection with the Integration run time details that is currently setup in the higher environment, post the deployment
Wasim 20 Reputation points

2024-02-06T04:50:41.89+00:00

Thank for responding, one more question, does it require of you to manually update the lined service connection with integration runtime each time you deploy from source env to target environment?
Sarthak 5 Reputation points

2024-02-09T11:28:59.68+00:00

yes we had to do that change every time we deploy to higher environment

Your answer

Sarthak 5 Reputation points

2023-12-13T05:53:03.1733333+00:00

Any help/suggestion for this issue.
AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator

2023-12-14T06:29:33.13+00:00

Hi Sarthak ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to deploy synapse workspace ARM template from Dev environment to UAT environment, however during this process you are experiencing an error "Could not figure out full dependency model for these artifacts" related to Integration runtime. Please let me know if that is not the case.

It seems that all the dependencies related to the IR properties are not exposed by default in the templateParametersForWorkspace.json and templateForWorkspace.json files.

Could you please try creating a custom parameter template which would include all the dependencies related to the IR like location computeType, coreCount, TimeToLive, etc.

For more details on how to use CustomParameter template , kindly checkout the following resources:

Synapse CI/CD: Using a custom parameter template

Create custom parameters in the workspace template

Hope it helps. Kindly let us know how it goes. Thankyou
Sarthak 5 Reputation points

2023-12-14T14:47:52.7666667+00:00

Hi Annu,

Thanks! for your response. I need your help in adding below mentioned properties to custom template parameter definition

"Could you please try creating a custom parameter template which would include all the dependencies related to the IR like location computeType, coreCount, TimeToLive, etc."

Because in the code definition of Self-Hosted IR i'm getting only these settings:

{ "name": "Integration-runtime-selfhost-eusfs02", "properties": { "type": "SelfHosted" } }

And here is the setup:

How do i include properties : location computeType, coreCount, TimeToLive to custom template parameter definition for integrationruntimes tag.

If you can please share some reference or links to do this custom setup that will help in my synapse pipeline deployment for Integrationruntimes to higher environment.
Sarthak 5 Reputation points

2023-12-18T08:32:24.04+00:00

Any update on my query AnnuKumari-MSFT
AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator

2023-12-19T09:10:22.45+00:00

Hi Sarthak ,

Kindly checkout the above video which has the end to end guidance on using custom parameter template. Sample json including the dependencies can be seen in the below image:

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou
Sarthak 5 Reputation points

2023-12-20T13:16:15.4266667+00:00

Hi,

I tried to use custom template-parameters-definition.json file for making entry Integration times

"Microsoft.Synapse/workspaces/integrationRuntimes": { "properties": { "typeProperties": { "*": "=" } } }

But in the template-parameter-defintion.json file here is the output i'm getting after i publish the code from main branch

"AutoResolveIntegrationRuntime_properties_typeProperties_computeProperties": { "value": { "location": "AutoResolve", "dataFlowProperties": { "computeType": "General", "coreCount": 8, "timeToLive": 0 } } }

There is no entry / reference for the self-hosted integration runtime

Integration-runtime-selfhost-eusfs02

.

Please let me know if i'm missing something here...Please advice
Sarthak 5 Reputation points

2023-12-26T07:24:48.0766667+00:00

@AnnuKumari-MSFT , Appreciate if you could please provide any guidance for this deployment issue..
AnnuKumari-MSFT 34,561 Reputation points Microsoft Employee Moderator

2023-12-26T08:43:44.3633333+00:00

Sarthak ,

Kindly check ARM template resource definition ,

{ "type": "Microsoft.DataFactory/factories/integrationRuntimes", "apiVersion": "2018-06-01", "name": "string", "properties": { "description": "string", "type": "string" // For remaining properties, see IntegrationRuntime objects } }

For properties of IntegrationRuntime objects, Set the type property to specify the type of object.

For SelfHosted, use:

"type": "SelfHosted", "typeProperties": { "linkedInfo": { "authorizationType": "string" // For remaining properties, see LinkedIntegrationRuntimeType objects } }

Hope it helps. Thankyou
Sarthak 5 Reputation points

2024-01-02T10:37:49.57+00:00

Here is the updated section for Integration runtime in file: template-parameters-definition.json ( as per the documentation and steps defined for SHIR)

"Microsoft.Synapse/workspaces/integrationRuntimes": { "name": "=", "properties": { "type": "SelfHosted", "typeProperties": { "linkedInfo": { "authorizationType": "key" } } } }

And after i publish from the main branch , here is the output captured in file: TemplateParametersForWorkspace.json

"AutoResolveIntegrationRuntime_name": { "value": "[concat(parameters('workspaceName'), '/AutoResolveIntegrationRuntime')]" }, "Integration-runtime-selfhost-eusfs02_name": { "value": "[concat(parameters('workspaceName'), '/Integration-runtime-selfhost-eusfs02')]" }

This wont help in the deployment through DevOps pipeline, i need to set the correct Integration runtime reference in the Linked service connection again at higher environment and republish

Any other suggestion or am i missing something here..
Sarthak 5 Reputation points

2024-01-08T13:32:00.2133333+00:00

Hi @AnnuKumari-MSFT please provide your input / suggestions. I tried to implement those steps (mentioned here in this link :https://learn.microsoft.com/en-us/azure/templates/microsoft.datafactory/factories/integrationruntimes?pivots=deployment-language-arm-template#resource-format-1. but it didn't work.
Wasim 20 Reputation points

2024-02-01T23:54:34.8166667+00:00

HI @sarthak , were you able to resolve the issue? I am faced with same issue, wonder how you resolved if you have already.
Sarthak 5 Reputation points

2024-02-05T16:42:25.8033333+00:00

Issue is still there, I had to manually update the Linked service connection with the Integration run time details that is currently setup in the higher environment, post the deployment
Wasim 20 Reputation points

2024-02-06T04:50:41.89+00:00

Thank for responding, one more question, does it require of you to manually update the lined service connection with integration runtime each time you deploy from source env to target environment?
Sarthak 5 Reputation points

2024-02-09T11:28:59.68+00:00

yes we had to do that change every time we deploy to higher environment

Share via

Azure Synapse Analytics- Overriding SelfHost IR Parameter

Your answer