Say I have some small files and some very large files all being processed via Synapse pipeline calls to Spark... how do I say the small files should run on a small cluster and the very large files on a bigger cluster? There does not seem to be much available on where spark sessions should run

Hello @Ryan Abbey , Thanks for the question and using MS Q&A platform. Unfortunately, there is no built-in mechanism to prioritize the jobs based on the file sizes. Azure Synapse provides this feature out of box in Apache Spark pools. Apache Spark pools provide the ability to automatically scale up and down compute resources based on the amount of activity . When the autoscale feature is enabled , you can set the minimum and maximum number of nodes to scale. When the autoscale feature is disabled , the number of nodes set will remain fixed. For more details, refer to Apache Spark pool configurations in Azure Synapse Analytics and Automatically scale Azure Synapse Analytics Apache Spark pools . Hope this helps. Do let us know if you any further queries. --------------------------------------------------------------------------- Please " Accept the answer " if the information helped you. This will help us and others in the community as well.

Synapse Spark pipeline - choosing cluster

Accepted answer

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-01T08:56:13.197+00:00
Hello @Ryan Abbey ,

Thanks for the question and using MS Q&A platform.

Unfortunately, there is no built-in mechanism to prioritize the jobs based on the file sizes.

Azure Synapse provides this feature out of box in Apache Spark pools.

Apache Spark pools provide the ability to automatically scale up and down compute resources based on the amount of activity.

When the autoscale feature is enabled, you can set the minimum and maximum number of nodes to scale.

When the autoscale feature is disabled, the number of nodes set will remain fixed.

For more details, refer to Apache Spark pool configurations in Azure Synapse Analytics and Automatically scale Azure Synapse Analytics Apache Spark pools.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.
Ryan Abbey 1,181 Reputation points

2021-09-01T21:10:42.9+00:00

Apache Spark pools is what we configure, correct? Currently I have three pools defined, how do I direct the pipeline to use a specific pool?

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-06T09:01:52.82+00:00

Hello @Ryan Abbey ,

Every activity in Synapse Pipeline is associated with the anyone of the Apache Spark Pool.

Example: If you select any Synapse Notebook/Spark Job definition is associated with the Apache Spark Pool.

Ryan Abbey 1,181 Reputation points

2021-09-07T04:17:23.433+00:00

I have no idea what that means, sorry... I have 3 spark pools, can I choose, from the pipeline, which of those the notebook should run against? If so, how?

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-08T10:05:09.057+00:00

Hello @Ryan Abbey ,

Every activity in Synapse Pipeline is associated with the anyone of the Apache Spark Pool.

For Spark Job Definition, you will find the settings tab where you have a option to choose the spark pools preconfigured.

For Notebooks, you can use the same spark pool which is already preconfigured.

Hope this helps. Do let us know if you any further queries.

Ryan Abbey 1,181 Reputation points

2021-09-08T19:52:28.517+00:00

Hmm...ok... so effectively it's going to use the cluster that was set during development?

What happens on a deployment and the clusters are named differently? Is there a way to alter this within the process or do we have to open the notebook within production and change?

Or if we want to change the cluster it uses within a production environment (e.g. data has grown exponentially and we need a bigger cluster)?

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-10T08:58:32.147+00:00

Hello @Ryan Abbey ,

Yes, it uses the same cluster that was set during development.

What happens on a deployment and the clusters are named differently? Is there a way to alter this within the process or do we have to open the notebook within production and change?

Example: Here I had created a pipeline to run a notebook 2 which has no cluster attached.

When I run this pipeline it will fail with the below error:

if we want to change the cluster it uses within a production environment (e.g. data has grown exponentially and we need a bigger cluster)?

You can use autoscale feature is enabled, you can set the minimum and maximum number of nodes to scale.

Apache Spark pools provide the ability to automatically scale up and down compute resources based on the amount of activity.

Hope this helps. Do let us know if you any further queries.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Ryan Abbey 1,181 Reputation points

2021-09-12T18:58:54.923+00:00

I wouldn't want spark suddenly deciding 50 nodes was required to process a 10MB file (let's face it, Microsoft has a long history of getting things wrong so why would I want to leave Microsoft in control of how much resources should be applied?)... I want the control to say, these small files go to a small cluster, these big files go to a big cluster and subsequently, this small file has now grown larger, I want to shift it to a big cluster...

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-13T05:54:45.897+00:00

Hello @Ryan Abbey ,

Unfortunately, there is no built-in mechanism to prioritize the jobs based on the file sizes.

We will share and elevate your suggestion with the Synapse product team.

PRADEEPCHEEKATLA 90,266 Reputation points

2021-09-15T09:00:56.93+00:00

Hello @Ryan Abbey ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Ryan Abbey 1,181 Reputation points

2021-09-26T20:25:00.26+00:00

I struggle to accept this as an answer since it feels more of a workaround... but... it currently is the way a cluster is selected, hopefully Microsoft will allow greater configurability in the future!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Synapse Spark pipeline - choosing cluster

0 additional answers

Your answer