All Diagnostics and Admin jobs Failed To Run but application can submit Job normally on HPC Pack 2012 R2 Cluster Manager.

Question

Hi All,

We are using HPC Pack 2012 R2 Update 3 on Failover Cluster.

Recently HPC Cluster Manager cannot run diagnostics and admin jobs like using run command in Cluster Manager.

But weirdly Cluster Manager can submit job normally.

Below is the log file of Diagnostics when an error occurs.

09/25/2020 04:46:29.147 i HpcDiagnostics 1896 14432 [Store] New test run submitted for test Id#4 by [Domain Account]
09/25/2020 04:46:29.148 i HpcDiagnostics 1896 14432 [Store] new RunId is #1419
09/25/2020 04:46:29.148 i HpcDiagnostics 1896 14432 [Store] parameter: -count:4
09/25/2020 04:46:29.148 i HpcDiagnostics 1896 14432 [Store] parameter: -Network:Default
09/25/2020 04:46:29.154 e HpcDiagnostics 1896 4752 An unexpected exception occurred. For more information about this exception, see the Details tab. . Additional data:. End of Stream encountered before parsing was completed. . Exception detail: System.Runtime.Serialization.SerializationException: End of Stream encountered before parsing was completed.....Server stack trace: .. at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run().. at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage).. at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage).. at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, IMethodCallMessage methodCallMessage).. at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)....Exception rethrown at [0]: .. at Microsoft.Hpc.Scheduler.NodeManagement.NodeQuery.HandleException(Exception e).. at Microsoft.Hpc.Scheduler.NodeManagement.NodeQuery.QueryNodes(IEnumerable`1 constraints).. at Microsoft.Hpc.Diagnostics.Controller.Utilities.GetNodesListFromNodeTypes(INodeQuery nodeQuery, DiagnosticTargetNodeType type).. at Microsoft.Hpc.Diagnostics.Controller.SubmittedTestHandler.FilterOutNotSupportedNodes(DiagnosticTestRun testRun, DiagnosticTest testDef).. at Microsoft.Hpc.Diagnostics.Controller.SubmittedTestHandler.ExecuteInternal(DiagnosticTestRun testRun).. at Microsoft.Hpc.Diagnosti� �� oller.StateHandlerBase.Execute()�)�; ��R�; t Mi nostics. 57v�� y, Excep eption).h�)�; .Hpc.Diagnostics.Controller. dler� cute().. crosoft.Hpc.Diagnostics.Controller.DiagnosticsController.RunStateHandlers(Object o).. at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx).. at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx).. at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem().. at System.Threading.ThreadPoolWorkQueue.Dispatch()..

I tried restart the HPC Diagnostics service, but the results were the same.

I would appreciate any help with this problem.
Thanks.

Answer 1

Hi,

HPC is currently not supported in the Q&A forums, the supported products are listed over here https://learn.microsoft.com/en-us/answers/products (more to be added later on).

You can ask the experts in the dedicated HPC forum over here:
https://social.microsoft.com/Forums/en-US/home?category=windowshpc

----------

(If the reply was helpful please don't forget to upvote or accept as answer, thank you)

Best regards,
Leon