Spark startup time
Ryan Abbey
1,181
Reputation points
Time to initialise our Spark application is now approaching 10 minutes! That is ridiculously slow, what can we look at to improve this time?
It's a small pool - and while it's configured to allow up to 80 nodes, the jobs we run are relatively small but we run many at once - and while running many at once seems to suggest the cause, it shouldn't since each job will only need the 3 nodes (and certainly only 3 at initialisation)
Sign in to answer