How to parameterize Azure Databricks Delta lake in DataFactory ?

Question

Hello,

I have a dev DataFactory with LinkedService (Azure Databricks Delta Lake, Azure Databricks, Azure Key Vault). I managed to parameterize the last 2, but didn't find a way to do ti for Azure Databricks Delta Lake.

So whan I publish the dev DF, it generated a ARMTemplateForFactory.json with static domain and clusterId :

	{  
		"name": "[concat(parameters('factoryName'), '/AzureDatabricksDeltaLake')]",  
		"type": "Microsoft.DataFactory/factories/linkedServices",  
		"apiVersion": "2018-06-01",  
		"properties": {  
			"annotations": [],  
			"type": "AzureDatabricksDeltaLake",  
			"typeProperties": {  
				"domain": "https://adb-XXX.azuredatabricks.net",  
				"clusterId": "YYY",  
				"accessToken": {  
					"type": "AzureKeyVaultSecret",  
					"store": {  
						"referenceName": "myKV",  
						"type": "LinkedServiceReference"  
					},  
					"secretName": "DatabricksAccessToken"  
				}  
			}  
		},  
		"dependsOn": [  
			"[concat(variables('factoryId'), '/linkedServices/myKV')]"  
		]  
	},  

In the 2 other linked services, I can put parameters :

			"parameters": {  
				"DatabricksUrl": {  
					"type": "string",  
					"defaultValue": "https://adb-XXX.azuredatabricks.net"  
				},  
				"ClusterId": {  
					"type": "string",  
					"defaultValue": "YYY"  
				}  
			},  

How to achieve this with Azure Databricks Delta lake linked service and be able to deploy this to my production DataFactory with the right domain and clusterId ?

Thanks.

Answer 1

Hi @MichalLeneveut-8304 ,

Not all connector linked services are supported parameterization using UI. Click here to see list of connectors which supports parameterization in UI

In your case its Azure Databricks Delta Lake connector. As per above documentation it's not supports parameterization using UI. Hence, we need to do parameterization using Advanced section Specify dynamic contents in JSON format.

Json used in above screenshot.

{  
    "properties": {  
        "type": "AzureDatabricksDeltaLake",  
        "parameters": {  
            "clusterId": {  
                "type": "String"  
            },  
            "accessToken": {  
                "type": "String"  
            }  
        },  
        "typeProperties": {  
            "domain": "https://<databricksDomain>.azuredatabricks.net",  
            "clusterId": "@{linkedService().clusterId}",  
            "accessToken": {  
                "type": "SecureString",  
                "value": "@{linkedService().accessToken}"  
            }  
        },  
        "annotations": []  
    }  
}  

Hope this helps. Please let us know if any further queries.

-------------------

Please consider hitting Accept Answer button. Accepted answers help community as well.