long creation time for concurrent sessions

Question

I'm noticing longer Spark session creation times (over 25 minutes) for notebooks running concurrently in my pipelines. These times seem to decrease as fewer notebooks are left to execute, reaching under a minute for the final one. Could the initial delay be due to the large number of Spark sessions being created simultaneously?

Answer 1

Hi @HideyoshiAkaio

Thanks for the question and using MS Q&A platform.

Based on your description, it is possible that the longer Spark session creation times are due to the large number of Spark sessions being created simultaneously. To provide more relevant information, could you please share a few screenshots of the pipeline that has multiple notebooks? This will help us better understand your scenario and provide more specific guidance.

In general, when you connect to a Spark pool, create a session, and run a job, a new Spark instance is created. As multiple users may have access to a single Spark pool, a new Spark instance is created for each user that connects. When you submit a second job, if there is capacity in the pool, the existing Spark instance also has capacity. Then, the existing instance will process the job. Otherwise, if capacity is available at the pool level, then a new Spark instance will be created.

Here are a few example scenarios related to Synapse Spark pool allocation with respect to notebook job executions that might give you more clarity: Apache Spark in Azure Synapse Analytics Core Concepts

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.