Collecting custom JSON data sources with the Log Analytics agent for Linux in Azure Monitor

Note

As part of the ongoing transition from Microsoft Operations Management Suite to Azure Monitor, the Operations Management Suite Agent for Windows or Linux will be referred to as the Log Analytics agent for Windows and Log Analytics agent for Linux.

Custom JSON data sources can be collected into Azure Monitor using the Log Analytics agent for Linux. These custom data sources can be simple scripts returning JSON such as curl or one of FluentD's 300+ plugins. This article describes the configuration required for this data collection.

Note

Log Analytics agent for Linux v1.1.0-217+ is required for Custom JSON Data. This collection flow only works with MMA. Consider moving to the AMA agent and using the additional collection features available there

Configuration

Configure input plugin

To collect JSON data in Azure Monitor, add oms.api. to the start of a FluentD tag in an input plugin.

For example, following is a separate configuration file exec-json.conf in /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.d/. This uses the FluentD plugin exec to run a curl command every 30 seconds. The output from this command is collected by the JSON output plugin.

<source>
  type exec
  command 'curl localhost/json.output'
  format json
  tag oms.api.httpresponse
  run_interval 30s
</source>

<match oms.api.httpresponse>
  type out_oms_api
  log_level info

  buffer_chunk_limit 5m
  buffer_type file
  buffer_path /var/opt/microsoft/omsagent/<workspace id>/state/out_oms_api_httpresponse*.buffer
  buffer_queue_limit 10
  flush_interval 20s
  retry_limit 10
  retry_wait 30s
</match>

The configuration file added under /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.d/ will require to have its ownership changed with the following command.

sudo chown omsagent:omiusers /etc/opt/microsoft/omsagent/conf/omsagent.d/exec-json.conf

Configure output plugin

Add the following output plugin configuration to the main configuration in /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.conf or as a separate configuration file placed in /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.d/

<match oms.api.**>
  type out_oms_api
  log_level info

  buffer_chunk_limit 5m
  buffer_type file
  buffer_path /var/opt/microsoft/omsagent/<workspace id>/state/out_oms_api*.buffer
  buffer_queue_limit 10
  flush_interval 20s
  retry_limit 10
  retry_wait 30s
</match>

Restart Log Analytics agent for Linux

Restart the Log Analytics agent for Linux service with the following command.

sudo /opt/microsoft/omsagent/bin/service_control restart 

Output

The data will be collected in Azure Monitor with a record type of <FLUENTD_TAG>_CL.

For example, the custom tag tag oms.api.tomcat in Azure Monitor with a record type of tomcat_CL. You could retrieve all records of this type with the following log query.

Type=tomcat_CL

Nested JSON data sources are supported, but are indexed based off of parent field. For example, the following JSON data is returned from a log query as tag_s : "[{ "a":"1", "b":"2" }].

{
    "tag": [{
      "a":"1",
      "b":"2"
    }]
}

Next steps

• Learn about log queries to analyze the data collected from data sources and solutions.