Synapse Spark Batch Stuck at Queued Status; Spark Pool Will Not Scale for Livy

Question

I'm using synapse spark pools, via the Livy Batch API:

https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/spark-batch

I've seen my cluster grow and shrink when an individual job needs more capacity as it is running. That "autoscale" functionality seems to work well in certain scenarios, and will scale the workers as specified (eg. from 3 nodes min to 6 nodes max ).

However, I'm having trouble with a seemingly less sophisticated scenario. Suppose I just need a whole bunch of small batch jobs to run, and consume a specified/predictable number of resources (2 driver vcores, 2 vcores each on 2 workers). If I submit a bunch of these at once, it will start running the ones that fit within initial 3 nodes of the cluster (the min number of nodes, as specified in the autoscale configuation). Once I reach the limit of batches that can be fit inside of 3 nodes, it will NOT start running any additional batch jobs. It will not scale the cluster up to 6 nodes. All my Livy batch jobs simply queue up and wait for available capacity on the minimal number of nodes (3 nodes).

I'm assuming that the "autoscale" functionality is managed by the spark cluster itself. I'm assuming that Livy has no direct influence on the behavior of autoscale. It seems likely that the spark cluster is not even aware of any of the queued jobs - since Livy is holding them back. So despite the fact that my cluster is designed to grow to 6 nodes, that does not help me where my Livy batch jobs are concerned.

Is there any way to encourage Livy to run my batch jobs, even after the minimal number of nodes (3) are at capacity? Is there any API to explicitly "autoscale" the cluster up to 6 nodes, if Livy isn't really smart enough to do that on its own authority? I haven't found the necessary API's or tools to influence my spark pool. I can't even find a way to start or stop a pool, or monitor the number of vcores, so I suspect that it is unlikely there is an API to scale it when needed. This seems to be a pretty unhelpful limitation of Livy...

Any ideas would be greatly appreciated. One thought that came to mind is possibly to send the pool a fake job that artificially consumes a ton of resources until the "autoscale" functionality is triggered. Does anyone have a sample that might serve this purpose? Would it be easier to influence autoscale by consuming large amounts of CPU, or large amounts of RAM? This idea (a greedy job that triggers scaling) assumes that Livy will be aware of the additional nodes once the spark pool has scaled up. Is it reasonable to assume that Livy will start using the additional nodes?

Answer 1

Same issue here. @MartinJaffer-MSFT any updates?

Answer 2

It has been a couple years, but I just wanted to post an update , now that I finally opened a CSS case.

There was some sort of "rounding" behavior where the calculations of vcores used by executors and drivers were only working properly for multiples of 4!

On 9/27/2023 I received the following update from CSS. I wish I could share the ICM # or BUG # but those are fairly hard to come by. I believe the "PG" which is mentioned here is referring to the "jobs-service" engineers within the Synapse Spark team.

"We have an update from the PG team that the microservice responsible for rounding off the cores to the nearest available size has been modified to accommodate smaller container sizes, we have also deployed the new bits and currently the release has reached the east us region, so it will be complete shortly and you will be able to see the improvements."

To make a long story short, it is possible that Livy will start behaving better, when submitting jobs for arbitrarily-sized executors. It is also possible that this will cause the spark pool to auto-size up to the max number of nodes (via Yarn). I am not holding my breath, until I see it happening myself. It is surprising to me that this problem wasn't reported & fixed a long time ago by a larger customer. .... If the problem is specific to me, then I'm not sure how I unwittingly bumped into it. (Is "4" a magical configuration for vcores, which everyone else seems to have agreed upon without telling me???)