Azure Databricks - Data Factory and Queueing of Jobs

Gopinath Rajee 651 Reputation points
2022-04-28T13:51:23.603+00:00

All,

I created a Pool with10 as the Max Capacity and I was hoping that the jobs submitted by all the ADF pipelines for which there is no instances left will queue up and wait for instances to be freed up. Unfortunately, the jobs are failing instead of waiting in the queue. Is there a workaround to ensure that jobs do queue up not fail?

Thanks,
grajee

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,071 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 85,351 Reputation points Microsoft Employee
    2022-05-02T05:40:10.31+00:00

    Hello @Gopinath Rajee ,

    Thanks for the question and using MS Q&A platform.

    As per error message which you have shared, your job required 1 more instance to successfully run the job.

    To refrain this, you can increase the capacity of the pool to some higher value as below:

    Go to Clusters > Pools > edit pool and increase the 'Max capacity' value.

    For more details, please refer : https://learn.microsoft.com/en-us/azure/databricks/clusters/instance-pools/configure#--maximum-capacity

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators