ADF & DataFlows

Question

All,

We plan to make use of Azure Data Factory's dataflows feature to handle huge amount of data and I'm aware that ADF deploys databricks to handle dataflows.

The question is, where does it create the VMs that are used by databricks? I looked at the Resource Group that contains the ADF and I don't see any resources created it in it for databricks.

Can someone point me in the right direction?

There is an elaborate thread here but I want to know if it creates any resources on the Resource Group.

https://learn.microsoft.com/en-us/answers/questions/678185/use-dedicated-cluster-in-adf-dataflows.html

Thanks,
Gopi

Answer 1

Hi @grajee ,

Thank you for posting query in Microsoft Q&A Platform.

ADF actually create serverless spark clusters to execute your dataflows. ADF will not create data bricks accounts to handle dataflows.

These serverless spark clusters will not be part of your resources or your resource groups. Its completely handled by Azure itself in backend. Hence you cannot see any resources or resource groups for them.

Hope this will help. Please let us know if any further queries.

---------------------------

Please consider hitting Accept Answer. Accepted answers helps community as well.