isolated databricks cluster call from synapses or azure datafactory

Question

how can I create a job in databricks with parameters of isolated from synapses or azure datafactory, because I can not find any option that allows to pass as parameter this value and not being able to do so I have no access to my unit catalog in databricks

Answer 1

Can you confirm if you have tried the following:

1. Create a Linked Service:
   • Click on + New and choose Azure Databricks as the compute option.
   • Fill in the required fields, including the Databricks workspace and select New job cluster to ensure isolation. This allows you to run jobs in an isolated manner without affecting shared clusters.
2. Configure the Databricks Notebook Activity:
   • Within your pipeline, add the Databricks Notebook Activity.
   • In the properties pane, you can specify parameters to be passed to the Databricks notebook. For example, you can define a parameter named input and set its value using an expression like @pipeline().parameters.name to pass values dynamically from ADF to Databricks.
3. Trigger and Monitor the Pipeline:
4. Access Unity Catalog:
   • Ensure that the Databricks workspace is configured to allow access to the Unity Catalog. This may require setting appropriate permissions and roles within Databricks to access the catalog from your isolated jobs.

Note:

• Cluster Isolation is generally managed within Databricks and is tied to the cluster configuration rather than being a parameter passed at runtime from ADF or Synapse.
• If you cannot directly pass the isolation as a parameter from ADF or Synapse, you need to ensure the job in Databricks is always associated with the isolated cluster that has access to Unity Catalog.