Notebook clusters in pipelines

Question

Notebook clusters in pipelines

Iwan 65

I have a pipeline which runs sql queries and copies the output into a parquet file, then using the output in a notebook activity.

It takes 3 minutes to run the sql queries and then 5 minutes for the cluster to start, then 2 minutes to run the notebook.

Is it possible to start the cluster in the pipeline in advance so I don't have to wait so long before running the notebook?

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Smaran Thoomu 32,265 Microsoft External Staff Moderator

Hi @Iwan

Welcome to the Microsoft Q&A and thank you for posting your questions here.

As I understand you're facing a common issue where the notebook activity in your pipeline is delayed due to the time it takes for the cluster to start.

Currently, in Azure Synapse Pipelines, there is no direct way to pre-warm or start the Spark cluster before reaching the notebook activity step.

However, a workaround is possible to start the cluster in advance within your pipeline to reduce the waiting time before running your notebook activity. Here’s a workaround to achieve this:

Insert a dummy notebook activity at the start of your pipeline that will effectively start the cluster. This notebook can be very simple, such as running a basic command like print("Pre-warming cluster"). The idea is to make sure the cluster is up and running by the time your main notebook activity is needed.
Add a Wait activity after the dummy notebook to ensure the cluster has enough time to start up. You can set this delay to the approximate time it takes for the cluster to start (e.g., 5 minutes).
After the Wait activity, proceed with your existing activities (running SQL queries, copying output to a Parquet file, running the main notebook activity).

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Iwan 65 Reputation points

2024-10-22T13:43:16.47+00:00

Thanks Smaran. Does this mean that if there are two notebook activities in the same pipeline execution using the same pool cluster, then the second notebook activity does not need to start up the pool again and continues where the first notebook left it?
Smaran Thoomu 32,265 Reputation points Microsoft External Staff Moderator

2024-10-22T14:19:33.9166667+00:00

Iwan Yes, if you have multiple notebook activities in the same pipeline using the same Spark pool, the cluster will stay alive between activities. So, the second notebook won't need to go through the full startup process again if it runs shortly after the first one.

This means that the cluster remains active, and you can continue running tasks on it without the startup delay, as long as the cluster hasn’t been decommissioned between the activities.
Iwan 65 Reputation points

2024-10-22T14:44:55.3133333+00:00

I've tested it now and it works as described, thanks
Smaran Thoomu 32,265 Reputation points Microsoft External Staff Moderator

2024-10-22T15:00:37.7333333+00:00

Iwan - Great to hear that it worked! I'm glad the solution helped. Feel free to continue using the MS Q&A forum for any further questions or if you need more assistance. We're always here to help!

Share via

Notebook clusters in pipelines

0 additional answers

Your answer