Azure Databricks - Data Factory and Queueing of Jobs

Question

Azure Databricks - Data Factory and Queueing of Jobs

Gopinath Rajee 656

All,

I created a Pool with10 as the Max Capacity and I was hoping that the jobs submitted by all the ADF pipelines for which there is no instances left will queue up and wait for instances to be freed up. Unfortunately, the jobs are failing instead of waiting in the queue. Is there a workaround to ensure that jobs do queue up not fail?

Thanks,
grajee

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-04-29T04:27:47.293+00:00

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

When you are saying jobs are failing rather than waiting in the queue, could you please share the error details on the failed jobs?
Gopinath Rajee 656 Reputation points

2022-04-29T11:56:32.14+00:00

Here is the error from databricks log:

Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster (2345-098981-j45cuk76)
to be ready.Cause Unexpected state for cluster (2345-098881-j45cuk76): INSTANCE_POOL_CLUSTER_FAILURE(CLIENT_ERROR):
instance_pool_error_code:INSTANCE_POOL_MAX_CAPACITY_FAILURE,instance_pool_id:0427-023128-cheep1-pool-1bx1mjds,
databricks_error_message:No more instances can be allocated from the pool because it reached its max capacity of 2,
but the cluster requires 1 instance(s). Try again later or increase the maximum capacity.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-05-05T07:14:06.883+00:00
Hello @Gopinath Rajee ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

1 answer

Your answer

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-04-29T04:27:47.293+00:00

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

When you are saying jobs are failing rather than waiting in the queue, could you please share the error details on the failed jobs?
Gopinath Rajee 656 Reputation points

2022-04-29T11:56:32.14+00:00

Here is the error from databricks log:

Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster (2345-098981-j45cuk76)
to be ready.Cause Unexpected state for cluster (2345-098881-j45cuk76): INSTANCE_POOL_CLUSTER_FAILURE(CLIENT_ERROR):
instance_pool_error_code:INSTANCE_POOL_MAX_CAPACITY_FAILURE,instance_pool_id:0427-023128-cheep1-pool-1bx1mjds,
databricks_error_message:No more instances can be allocated from the pool because it reached its max capacity of 2,
but the cluster requires 1 instance(s). Try again later or increase the maximum capacity.
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-05-05T07:14:06.883+00:00

Hello @Gopinath Rajee ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

Answer 1

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

As per error message which you have shared, your job required 1 more instance to successfully run the job.

To refrain this, you can increase the capacity of the pool to some higher value as below:

Go to Clusters > Pools > edit pool and increase the 'Max capacity' value.

For more details, please refer : https://learn.microsoft.com/en-us/azure/databricks/clusters/instance-pools/configure#--maximum-capacity

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2022-05-10T06:48:15.587+00:00

Hello @Gopinath Rajee ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Share via

Azure Databricks - Data Factory and Queueing of Jobs

1 answer

Your answer