Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,185 questions
HPC Pack 2019: slowness on the scheduler due to the DBs
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 67%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Alberto Garcia Fernandez
26
Reputation points
Hi everyone,
We write here in order to get some insight of what's happening to our Windows HPC cluster. We have an on-premises cluster of Windows Server 2019 servers, with 3 head nodes and 136 compute nodes, deploying a HPC Pack 2019 with built'in HA and the DBs hosted externally on always-on instances.
We find that the job validation takes too long (can be 5 minutes for the simplest of the tasks) although it gets eventually submitted. On the logs we find exceptions like the one below. And it seems that simple scheduler queries takes tens of seconds to be performed. But we have checked the DBs and don't seem to be overloaded on CPU, memory, or disk. All HPC Pack DBs (Management, Scheduler, Monitoring, Reporting, etc.) are hosted on a 16 vCPU and 64 GB RAM instance (which is replicated in always on mode).
- Do you think in your experience this size is appropriate ?
- Do you think of any other parameter that may be hindering the queries?
Any help or tip would be greatly appreciated.
Best Regards,
Alberto.
at Microsoft.Hpc.Scheduler.Store.LocalRowEnumerator.SetProperties(StoreProperty[] properties)..
at Microsoft.Hpc.Scheduler.JobValidatorNew.SingleThread.JobValidatorSingleThread.validateTaskChanges(Boolean expandParametric)..
at Microsoft.Hpc.Scheduler.JobValidatorNew.SingleThread.JobValidatorSingleThread.checkAndValidate()..
at Microsoft.Hpc.Scheduler.JobValidatorNew.SingleThread.JobValidatorSingleThread.validateThreadMain()
12/04/2020 07:48:12.842 w HpcScheduler 6560 3568 [Policy] validateThreadMain in JobValidatorSingleThread suffered an exception, but will retry: Microsoft.Hpc.Scheduler.Properties.SchedulerException: The scheduler server is busy. It cannot handle the client request now. Please try again later. For detailed information please check the scheduler event log on head node. ex System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out..
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction).. at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose).. at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady).. at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async, Int32 timeout, Boolean asyncWrite).. at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)..
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()..
################ -
Alberto García • HPC Architect
alberto.garcia@hpcnow.com
www.hpcnow.com
Follow us on Twitter • Linkedin
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 78%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECZ%3C/text%3E%3C/svg%3E)
Cris Zhan-MSFT 6,606 Reputation points2020-12-07T10:12:09.67+00:00 Hi AlbertoGarciaFernandez-3057,
Not familiar with HPC Pack, but you mean that the SQL Server instance hosting these HPC Pack DBs is the Always On Failover Cluster Instances deployed locally?
Also check this doc-Database Capacity Planning and Tuning for Microsoft HPC Pack for reference. -
Alberto Garcia Fernandez 26 Reputation points
2020-12-08T17:48:55.617+00:00 Thanks for your answer. We managed to solve the problem.
DBs are hosted on a different infrastructure, on an always on cluster. The problem we had was that the default timeout in the DB's connection strings was too short. One of our users sent a 1M task job, creating a giant query that couldn't get resolved before timeout. HPC Pack continuously resubmitted the query thus collapsing the Scheduler DB.
Increasing the timeout solved the problem.
Thanks in any case for your answer.
Best Regards,
Alberto. -
Cris Zhan-MSFT 6,606 Reputation points
2020-12-09T01:17:43.31+00:00 I am glad to hear that you have solved your problem.
Thank you for sharing.
Best Regards, -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 6%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Casey Averill 1 Reputation point2024-01-10T20:29:25.1266667+00:00 @Alberto Garcia Fernandez can you share the specific connection string/timeout values you used? We're facing a similar issue.
Sign in to comment