How to Add a subqueue in yarn

Vamsi Anamaneni 1

I already have queues setup on Yarn on HdInsight, they were setup with the Ambari UI.

I have a queue for sqoop that takes up 70% of the cluster.
However I have a few huge sqoop jobs with a lot of mappers that take up 100% of the queue and block other sqoop jobs from running.
I wanted to send all sqoop jobs greater than 512 sqoop mappers into a subqueue that had a max of 80% of the sqoop queue which would allow the smaller sqoop jobs to run concurrently and also the subqueue, and also allows the smaller sqoop job subqueue to use the resources of the larger sqoop job subqueue if theres resources available.

Is something like this possible?

I see subqueues in the yarn documentation: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
I don't see subqueues in Microsoft documentation: https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-troubleshoot-yarn

How do I setup the subqueues on Ambari UI on HDInsight?

HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2022-08-11T22:06:40.74+00:00

Hello @Vamsi Anamaneni ,
Thanks for the ask and using the Microsoft Q&A platform.
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.
Thanks
Himanshu
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2022-09-08T19:06:11.337+00:00

Hello @Vamsi Anamaneni ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.
Thanks
Himanshu

Share via

How to Add a subqueue in yarn

Your answer