Application Insights (java/springboot) missing CommandPoolBusyThreads and CommandPoolIdleThreads metrics

Question

I seem to be missing a metric that reports busy vs idle threads on my java app. This is a springboot app, integrated with App Insights by passing in the -javaagent parameter with the java command to launch the app.

Using the following KQL query, I only see these 8 metrics stored, none of which indicate active vs idle threads:

customMetrics
| summarize by name

Suspected Deadlocked Threads
Heap Memory Used (MB)
% Of Max Heap Memory Used
Loaded Class Count
GC Total Count
GC Total Time
Current Thread Count
HeartbeatState

If I look at https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported, I see Azure Monitor has support for both CommandPoolIdleThreads and CommandPoolBusyThreads, but those do not show up in the customMetrics table.

Is there something I can do to force it to record these additional metrics?

Related: is it possible for you to add an "application-insights" tag? I added the azure-metrics-advisor tag, however I'm not certain that is most optimal for this post.

Thanks

Answer 1

To follow up on this, I enabled debug mode via applicationinsights.json to see what other JMX metrics might be possible to capture and I found these:

2022-03-15 20:44:34.603-04:00 INFO  c.m.a.a.i.p.AvailableJmxMetricLogger - available jmx metrics:
  - object name:        java.lang:type=MemoryPool,name=PS Survivor Space
    attributes: CollectionUsage.committed (number), CollectionUsage.init (number), CollectionUsage.max (number), CollectionUsage.used (number), CollectionUsageThreshold (number), CollectionUsageThresholdCount (number), CollectionUsageThresholdExceeded (boolean), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (exception), UsageThresholdCount (exception), UsageThresholdExceeded (exception), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        java.lang:type=MemoryManager,name=Metaspace Manager
    attributes: MemoryPoolNames (other), Name (string), ObjectName (other), Valid (boolean)
  - object name:        java.lang:type=GarbageCollector,name=PS MarkSweep
    attributes: CollectionCount (number), CollectionTime (number), LastGcInfo.duration (number), LastGcInfo.endTime (number), LastGcInfo.id (number), LastGcInfo.memoryUsageAfterGc (other), LastGcInfo.memoryUsageBeforeGc (other), LastGcInfo.startTime (number), MemoryPoolNames (other), Name (string), ObjectName (other), Valid (boolean)
  - object name:        JMImplementation:type=MBeanServerDelegate
    attributes: ImplementationName (string), ImplementationVendor (string), ImplementationVersion (string), MBeanServerId (string), SpecificationName (string), SpecificationVendor (string), SpecificationVersion (number)
  - object name:        java.lang:type=Runtime
    attributes: BootClassPath (string), BootClassPathSupported (boolean), ClassPath (string), InputArguments (other), LibraryPath (string), ManagementSpecVersion (number), Name (string), ObjectName (other), SpecName (string), SpecVendor (string), SpecVersion (number), StartTime (number), SystemProperties (other), Uptime (number), VmName (string), VmVendor (string), VmVersion (string)
  - object name:        java.lang:type=Threading
    attributes: AllThreadIds (other), CurrentThreadCpuTime (number), CurrentThreadCpuTimeSupported (boolean), CurrentThreadUserTime (number), DaemonThreadCount (number), ObjectMonitorUsageSupported (boolean), ObjectName (other), PeakThreadCount (number), SynchronizerUsageSupported (boolean), ThreadAllocatedMemoryEnabled (boolean), ThreadAllocatedMemorySupported (boolean), ThreadContentionMonitoringEnabled (boolean), ThreadContentionMonitoringSupported (boolean), ThreadCount (number), ThreadCpuTimeEnabled (boolean), ThreadCpuTimeSupported (boolean), TotalStartedThreadCount (number)
  - object name:        java.lang:type=OperatingSystem
    attributes: Arch (string), AvailableProcessors (number), CommittedVirtualMemorySize (number), FreePhysicalMemorySize (number), FreeSwapSpaceSize (number), MaxFileDescriptorCount (number), Name (string), ObjectName (other), OpenFileDescriptorCount (number), ProcessCpuLoad (number), ProcessCpuTime (number), SystemCpuLoad (number), SystemLoadAverage (number), TotalPhysicalMemorySize (number), TotalSwapSpaceSize (number), Version (string)
  - object name:        java.lang:type=MemoryPool,name=Metaspace
    attributes: CollectionUsage (exception), CollectionUsageThreshold (exception), CollectionUsageThresholdCount (exception), CollectionUsageThresholdExceeded (exception), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (number), UsageThresholdCount (number), UsageThresholdExceeded (boolean), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        java.lang:type=Compilation
    attributes: CompilationTimeMonitoringSupported (boolean), Name (string), ObjectName (other), TotalCompilationTime (number)
  - object name:        java.lang:type=MemoryManager,name=CodeCacheManager
    attributes: MemoryPoolNames (other), Name (string), ObjectName (other), Valid (boolean)
  - object name:        java.lang:type=MemoryPool,name=Compressed Class Space
    attributes: CollectionUsage (exception), CollectionUsageThreshold (exception), CollectionUsageThresholdCount (exception), CollectionUsageThresholdExceeded (exception), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (number), UsageThresholdCount (number), UsageThresholdExceeded (boolean), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        java.nio:type=BufferPool,name=direct
    attributes: Count (number), MemoryUsed (number), Name (string), ObjectName (other), TotalCapacity (number)
  - object name:        java.lang:type=Memory
    attributes: HeapMemoryUsage.committed (number), HeapMemoryUsage.init (number), HeapMemoryUsage.max (number), HeapMemoryUsage.used (number), NonHeapMemoryUsage.committed (number), NonHeapMemoryUsage.init (number), NonHeapMemoryUsage.max (number), NonHeapMemoryUsage.used (number), ObjectName (other), ObjectPendingFinalizationCount (number), Verbose (boolean)
  - object name:        java.lang:type=GarbageCollector,name=PS Scavenge
    attributes: CollectionCount (number), CollectionTime (number), LastGcInfo.duration (number), LastGcInfo.endTime (number), LastGcInfo.id (number), LastGcInfo.memoryUsageAfterGc (other), LastGcInfo.memoryUsageBeforeGc (other), LastGcInfo.startTime (number), MemoryPoolNames (other), Name (string), ObjectName (other), Valid (boolean)
  - object name:        java.lang:type=MemoryPool,name=PS Old Gen
    attributes: CollectionUsage.committed (number), CollectionUsage.init (number), CollectionUsage.max (number), CollectionUsage.used (number), CollectionUsageThreshold (number), CollectionUsageThresholdCount (number), CollectionUsageThresholdExceeded (boolean), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (number), UsageThresholdCount (number), UsageThresholdExceeded (boolean), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        java.util.logging:type=Logging
    attributes: LoggerNames (other), ObjectName (other)
  - object name:        java.lang:type=MemoryPool,name=PS Eden Space
    attributes: CollectionUsage.committed (number), CollectionUsage.init (number), CollectionUsage.max (number), CollectionUsage.used (number), CollectionUsageThreshold (number), CollectionUsageThresholdCount (number), CollectionUsageThresholdExceeded (boolean), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (exception), UsageThresholdCount (exception), UsageThresholdExceeded (exception), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        java.lang:type=ClassLoading
    attributes: LoadedClassCount (number), ObjectName (other), TotalLoadedClassCount (number), UnloadedClassCount (number), Verbose (boolean)
  - object name:        java.lang:type=MemoryPool,name=Code Cache
    attributes: CollectionUsage (exception), CollectionUsageThreshold (exception), CollectionUsageThresholdCount (exception), CollectionUsageThresholdExceeded (exception), CollectionUsageThresholdSupported (boolean), MemoryManagerNames (other), Name (string), ObjectName (other), PeakUsage.committed (number), PeakUsage.init (number), PeakUsage.max (number), PeakUsage.used (number), Type (string), Usage.committed (number), Usage.init (number), Usage.max (number), Usage.used (number), UsageThreshold (number), UsageThresholdCount (number), UsageThresholdExceeded (boolean), UsageThresholdSupported (boolean), Valid (boolean)
  - object name:        com.sun.management:type=DiagnosticCommand
    attributes: (no attributes found)
  - object name:        com.sun.management:type=HotSpotDiagnostic
    attributes: DiagnosticOptions (other), ObjectName (other)
  - object name:        java.nio:type=BufferPool,name=mapped
    attributes: Count (number), MemoryUsed (number), Name (string), ObjectName (other), TotalCapacity (number)

My guess is the current "ThreadCount" it pulls is this: getThreadCount()Returns the current number of live threads including both daemon and non-daemon threads.

But maybe what would be more valuable is this: getDaemonThreadCount()Returns the current number of live daemon threads.

(from https://docs.oracle.com/javase/8/docs/api/java/lang/management/ThreadMXBean.html)

Although those are just guesses -- I would really like to know for certain which value has the number of active threads being used, and which one has the maxThreads available. These two values are useful for setting monitoring thresholds.

I think that should give us the actual active threads....

Answer 2

Hi @EJ Marmonti ,

Welcome to Microsoft Q&A! Thanks for posting the question.

I understand that you are trying to capture additional thread related metrics from your Java application using ApplicationInsights, apart form the ones which are being collected by default. Please find my response below regarding this query:

1. Please note that the CommandPoolIdleThreads and CommandPoolBusyThreads metric as mentioned in the referenced document are for "Microsoft.AnalysisServices/servers" and "Microsoft.PowerBIDedicated/capacities" Azure service respectively and not for "ApplicationInsights" itself.

2. Application Insights Java 3.x collects some of the JMX metrics by default however, if it is not enough, you may refer to Configuring JMX metrics link for details on how to collect additional metrics.

3. In case you would also want to collect additional custom metrics (probably the one generated or collected by your application), you may also use the telemetryClient.trackMetric() to send it to ApplicationInsights. ref: Send custom telemetry

The tables and columns available in ApplicationInsights are available at this link.

Please let me know if you have any questions.

---
Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.