hello, Is there a way to find out what spark cluster size each of my ADF environment is using? Thanks

Data flows run on Spark clusters that are spun up at run-time. The configuration for the cluster used is defined in the integration runtime (IR) of the activity. https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance https://kromerbigdata.com/2019/11/07/adf-data-flows-start-a-larger-debug-session/

cluster size used in ADF

Accepted answer

Nandan Hegde 31,511 Reputation points MVP

2023-05-11T08:41:45.78+00:00

Data flows run on Spark clusters that are spun up at run-time. The configuration for the cluster used is defined in the integration runtime (IR) of the activity.

https://learn.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime-performance

https://kromerbigdata.com/2019/11/07/adf-data-flows-start-a-larger-debug-session/
Please sign in to rate this answer.
arkiboys 9,686 Reputation points

2023-05-11T09:05:19.29+00:00

incorrect message

Nandan Hegde 31,511 Reputation points MVP

2023-05-11T09:22:24.7833333+00:00

Hey,

Sorry for the wrong aspect :)

Can you please clarify what is it that you require?
Basically the size of cluster you can define in the dataflow runtime settings. Are you asking whether there is an in built function to provide you the size in dynamic expression part ?
Sign in to comment

Share via

cluster size used in ADF

0 additional answers