Spark startup time

Ryan Abbey 1,171 Reputation points
2022-09-22T02:49:13.58+00:00

Time to initialise our Spark application is now approaching 10 minutes! That is ridiculously slow, what can we look at to improve this time?

243715-image.png

It's a small pool - and while it's configured to allow up to 80 nodes, the jobs we run are relatively small but we run many at once - and while running many at once seems to suggest the cause, it shouldn't since each job will only need the 3 nodes (and certainly only 3 at initialisation)

243696-image.png

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,442 questions
{count} votes