How to speed up start spark session

Question

In the integration module, I create a new pipeline to execute notebook. When multiple notebooks are executed in parallel (no more than 4 notebooks at the same time), it is found that when the pipeline starts the spark session, it takes a very long time to start, and sometimes even takes 17 hours to complete the start. Is there any way to automatically inherit the previously started spark session or reduce the startup time?Thanks in advance.

Answer 1

Hello @Liu, Su,

Welcome to the MS Q&A platform.

The reason for seeing a long time to start the spark instance is a capacity issue at your spark pool/instance level.

Spark instances:

Spark instances are created when you connect to a Spark pool, create a session, and run a job. As multiple users may have access to a single Spark pool, a new Spark instance is created for each user that connects.

When you submit a second job, if there is capacity in the pool, the existing Spark instance also has capacity. Then, the existing instance will process the job. Otherwise, if capacity is available at the pool level, then a new Spark instance will be created.

Please see the below example:

• You create a Spark pool called SP1; it has a fixed cluster size of 20 nodes.
• You submit a notebook job, J1 that uses 10 nodes, a Spark instance, SI1 is created to process the job.
• You now submit another job, J2, that uses 10 nodes because there is still capacity in the pool and the instance, the J2, is processed by SI1.
• If J2 had asked for 11 nodes, there would not have been capacity in SP1 or SI1. In this case, if J2 comes from a notebook, then the job will be rejected; if J2 comes from a batch job, then it will be queued.

In your case, when the 2nd, 3rd, and 4th notebooks run, if there is capacity in the same spark instance, then the same spark instance (instance used by notebook 1) will be used. Otherwise, the synapse spins up a separate spark instance.

When the jobs come as batch jobs, and there is no capacity in the spark instance, the job will be queued. (until the other spark instance releases the resources, this job will be queued). I believe this is causing your spark instance to wait a long time.

There are a couple more examples in the below document to better understand the spark instance/pool capacity.

https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-concepts#examples

In your case, please make sure the spark configuration is optimized for your spark cluster. Meaning the node size, number of nodes, number of executors etc

You can also set the executor size and number executor at the notebook level by going on the notebook configure session option.

As a best practice, it is advised to create smaller Spark pools that may be used for development and debugging and then larger ones for running production workloads.

When a Spark pool is created, it exists only as metadata, and no resources are consumed, running, or charged for. A Spark pool has a series of properties that control the characteristics of a Spark instance. These characteristics include but aren't limited to name, size, scaling behavior, time to live.

As there's no dollar or resource cost associated with creating Spark pools, any number can be created with any number of different configurations. Permissions can also be applied to Spark pools allowing users only to have access to some and not others.

I hope this helps. Please let me know if you have any further questions.

---

If this answers your question, please consider accepting the answer by hitting the Accept answer button, as it helps the community.