Unable to configure Azure Monitor OpenTelemetry-based auto-instrumentation for Java application running on Azure VM

Question

I am trying to configure Azure Monitor OpenTelemetry-based auto-instrumentation for Java application running on Azure VM. I have my application running on JBoss on one server and Pivotal Tc Server on another server. I tried to configure Azure monitor on both servers using the article below but not able to see anything on the Application Insights resources after configuring. My Java application is running on Azure vm but it's not running on Azure App Service. Based on the article below, I think I should be able to do a codeless monitor my application in Application Insights resource after configuring but I am not able to generate the applicationinsights.log in the same folder where I have placed applicationinsights-agent-3.2.8.jar and applicationinsights.json. Any help is appreciated.

https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#autocollected-requests-dependencies-logs-and-metrics

OS - Windows 2019 Datacenter
Java Application running on JBoss or Pivotal Tc Server.
JVM configured for JBoss domain or using tomcat setup for TcServer - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-arguments

Answer 1

@TejasShah-0975, thank you for the detailed response above. It helped me a lot to configure a test environment on an Azure Windows VM, to test with both servers and here are my findings/recommendations (I am summarizing the findings and step so that it would help others too looking for help on this topic):

For Azure Monitor OpenTelemetry-based auto-instrumentation for Java applications, the basic requirements are as below:

1. Point the JVM to the jar file

2. Set the Application Insights connection string

One of the most effective ways to troubleshoot issues related to it is to review the applicationinsights.log file, which is created in the applicationinsights**.jar directory. However, in current scenario, the .log file itself was not getting generated. This points to the fact that either the path mentioned is incorrect for the ApplicationInsights**.jar file OR that the supplied -javaagent parameter for JVM did not take into effect.

---
Here are my recommendations/findings for the 2 servers:

1. JBOSS EAP 7.3 on Windows VM

There are two operating modes for JBOSS servers - Standalone Server and Managed domain. I observed that you were modifying the host.xml file with -javaagent parameter, but it looks like you are using JBOSS in standalone mode. Therefore, please update the JBOSS_HOME\bin\standalone.conf.bat file to update the JAVA_OPTS environment variable, to also include the -javaagent parameter.

You may use Process Explorer to review the environment variables set for any process. Once the changes above were made, I noticed the following:

After verifying the above changes, I was able to get the applicationInsights.log file generated and verified that the data is getting ingested in the AppInsights workspace.

---
2. Pivotal TC Server

After going through the documentation and a couple of other reference materials, I came across this link - https://community.pivotal.io/s/article/Customizing-the-environment-or-JVM-options-for-vFabric-tc-Server?language=en_US

It states that - "When running tc Server as a service on Windows, JVM settings are specified in the conf/wrapper.conf file.".

The only change you need to make is to add wrapper.java.additional.X. with value as -javaagent... parameter, as shown below:

After making this change and restarting the tc server instance, the required JVM parameter for -javaagent was supplied to process as verified using Process Explorer.

Please note, -javaagent is used for instrumentation/profiling and the appinsights jar file reference must be provided using this switch only. The way this switch is provided to JVM will depend on the Server technology being used. For more information, you may reach out to the respective Server support team about the configuration details.

If you have any questions specific to ApplicationInsights, please let me know.

---
Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

Answer 2

@AnuragSingh-MSFT , yes both steps are performed. Any chance we can connect to help troubleshoot this.

Here are configurations for two different app servers we use. In neither of the application server configuration, applicationinsights.log is generating right now.

Article used - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-arguments

Application Server #1 JBoss - domain - Reference - Jboss EAP 7.0 - Domain server configuration (https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-arguments ).

• Point the JVM to the jar file.
  File C:\JBoss\jboss-eap-7.3\domain\configuration\host.xml
  <!--Add Java agent jar file here, tried both relative and absolute path for the jar file. Also placed the applicationinsight.json in the same folder as the jar file-->
  <!--option value="-javaagent:JBOSS_HOME\applicationinsights-agent-3.2.8.jar"/-->
  <option value="-javaagent:C:\JBoss\jboss-eap-7.3\applicationinsights-agent-3.2.8.jar"/> 2. Set the Application Insights connection string. ******applicationinsights.json
  {
  "connectionString": "InstrumentationKey=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://centralus-2.in.applicationinsights.azure.com/",
  "role": {
  "name": "QA-test-05"
  },
  "sampling": {
  "percentage": 100
  },
  "jmxMetrics": [
  ],
  "customDimensions": {
  },
  "instrumentation": {
  "logging": {
  "level": "INFO"
  },
  "micrometer": {
  "enabled": true
  }
  },
  "proxy": {
  },
  "preview": {
  "processors": [
  ]
  },
  "selfDiagnostics": {
  "destination": "file+console",
  "level": "INFO",
  "file": {
  "path": "C:\JBoss\jboss-eap-7.3\applicationinsights.log",
  "maxSizeMb": 5,
  "maxHistory": 1
  }
  }
  }

Application Server #2 Pivotal Tcserver Reference used - https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-arguments
https://tcserver.docs.pivotal.io/3x/docs-tcserver/topics/postinstall-getting-started.html#postinstall-startstop-tcserver-windows

1. Point the JVM to the jar file. location of the file -

File Java Class path - C:\Shaw\domain\Spectrum\conf\wrapper.conf
wrapper.java.classpath.5=%CATALINA_BASE%\bin\winx86_64\applicationinsights-agent-3.2.8.jar
File

• C:\Shaw\domain\Spectrum\bin\setenv.bat
  set JAVA_HOME=C:\Program Files\OpenJDK\openjdk-11.0.13_8
  set AGENT_PATHS=
  set "JAVA_AGENTS=%CATALINA_BASE%\bin\winx86_64\applicationinsights-agent-3.2.8.jar"
  set JAVA_LIBRARY_PATH=
  set JVM_OPTS=-Xmx512M -Xss256M
  set JDK_JAVA_OPTIONS="--add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED"
  set JAVA_OPTS=%JVM_OPTS% %AGENT_PATHS% %JAVA_AGENTS% %JAVA_LIBRARY_PATH%
  set "CATALINA_OPTS=%CATALINA_OPTS% -javaagent:%CATALINA_BASE%\bin\winx86_64\applicationinsights-agent-3.2.8.jar"
  File
• C:\pivotal-tc-server-standard-4.1.10.RELEASE\pivotal-tc-server\runtimes\tomcat-9.0.52.A.RELEASE\bin\catalina.bat
  rem Add on applicationinsights.jar file to CLASSPATH
  rem set "CLASSPATH=%CATALINA_HOME%\bin\applicationinsights-agent-3.2.8.jar"
  set "CLASSPATH=%CATALINA_BASE%\bin\winx86_64\applicationinsights-agent-3.2.8.jar"
  :applicationinsights-agent ClassPathDone
  File
• C:\pivotal-tc-server-standard-4.1.10.RELEASE\pivotal-tc-server\runtimes\tomcat-9.0.52.A.RELEASE\bin\tool-wrapper.bat
  set "CLASSPATH=%CLASSPATH%%CATALINA_HOME%\bin\bootstrap.jar;%CATALINA_HOME%\bin\tomcat-juli.jar;%CATALINA_HOME%\lib\servlet-api.jar;%CATALINA_HOME%\lib\tomcat-util.jar;%CATALINA_BASE%\bin\winx86_64\applicationinsights-agent-3.2.8.jar"

2. Set the Application Insights Connection string. File - C:\Shaw\domain\Spectrum\bin\winx86_64\applicationinsights.json {
   "connectionString": "InstrumentationKey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://centralus-2.in.applicationinsights.azure.com/",
   "role": {
   "name": "sample appspectrum test app"
   },
   "sampling": {
   "percentage": 100
   },
   "jmxMetrics": [
   ],
   "customDimensions": {
   },
   "instrumentation": {
   "logging": {
   "level": "TRACE"
   },
   }
   "selfDiagnostics": {
   "destination": "file+console",
   "level": "INFO",
   "file": {
   "path": "applicationinsights.log",
   "maxSizeMb": 5,
   "maxHistory": 1
   }
   }
   }