cluster size used in ADF

arkiboys 9,686 Reputation points
2023-05-11T08:03:32.92+00:00

hello,

Is there a way to find out what spark cluster size each of my ADF environment is using?
Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,915 questions
0 comments No comments
{count} votes

Accepted answer
  1. Nandan Hegde 30,081 Reputation points MVP
    2023-05-11T08:41:45.78+00:00

    Data flows run on Spark clusters that are spun up at run-time. The configuration for the cluster used is defined in the integration runtime (IR) of the activity.

    https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance

    https://kromerbigdata.com/2019/11/07/adf-data-flows-start-a-larger-debug-session/


0 additional answers

Sort by: Most helpful