Adding per-worker metrics reported to JMX in applicationinsights.json for App Insights

Ivan Webber 0 Microsoft Employee

I have a Flink pipeline and if possible I would like to report the metrics to JMX so that I can use the applicationinsights.json to report them to Application Insights. However, when Flink metrics are reported to JMX they have object names with generated IDs like below:

org.apache.flink.taskmanager.job.task.operator.currentOutputWatermark:host=,job_id=<ID>,job_name=Flink_metrics_test,operator_id=<ID>,operator_name=Sink-_logger-sink,subtask_index=0,task_attempt_id=<ID>,task_attempt_num=0,task_id=<ID>,task_name=Source-_Custom_Source_-_time-now-stream_-_counter-map_-_Sink-_logger-sink,tm_id=<ID>

Is it still possible to configure them as JMX metrics? Related documentation: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-jmx-metrics-configuration#configuration-example

Ivan Webber 0 Reputation points Microsoft Employee

2024-02-21T18:58:36.9866667+00:00

I can update the Flink reporter configuration so that the object names are more deterministic, but there would still be one object per parallel task (e.g. subtask_index=0, subtask_index=1, ...). We would like to be able to freely change the parallelism or auto-scale the job without having to rework our metrics or configuration, so we need a wildcard way to select all the beans in the domain and take the mininum per operator.
Monalla-MSFT 12,861 Reputation points

2024-02-28T19:21:12.17+00:00
@Ivan Webber - Thanks for reaching out to us.

Is the data not showing up or is it missing a dimension?

Is the data not structured how you are looking for?

What would you like to see in Azure Monitor?

What is it that you are unable to do in Azure portal today?

Please provide the requested information, so we can investigate this further. Thank you.
Ivan Webber 0 Reputation points Microsoft Employee

2024-03-05T22:39:05.4266667+00:00

Thanks Monalla. In our case we were able to get the data to show up to some extent, but it's not ideal. Flink's JMX reporter reports a domain for each metric with a different bean for each subtask (unit of parallelization generally mapping to one cpu core). Our current solution is to add a few thousand lines to the applicationinsights.json configuration so that all the JMX metrics for each parallel operator/task can be sent, but if the parallelism is changed or autoscaled above what we put in the configuration the metrics for those subtasks wouldn't show up. Ideally, we'd be able to specify the object name with a wildcard or regular expression and any matching JMX metrics would be sent to application insights with the actual object name available as a column or attribute. For you to better understand our use case see this Flink blog post which explains alerts on watermarks.
Monalla-MSFT 12,861 Reputation points

2024-03-08T17:23:37.2366667+00:00

@Ivan Webber - Sure, thank you for providing more details. I have raised an internal Github issue for internal PG team to look at :https://github.com/microsoft/ApplicationInsights-Java/issues/3582 You can please check the status in that GitHub issue.Thank you
Ivan Webber 0 Reputation points Microsoft Employee

2024-03-14T18:00:06.3866667+00:00

Thanks, I can't comment on the issue, but someone responded saying * symbols can be used in the object names in place of GUIDs. The documentation needs updated because it made no mention or example of that. Additionally, are these able to replace values in general (i.e. does it work like Regex pattern matching, so I could use a pattern like "objectName": "*.currentOutputWatermark",). Thanks for following up!
Monalla-MSFT 12,861 Reputation points

2024-03-18T17:27:41.42+00:00

@Ivan Webber - Sure, Iet me follow up on that and will get back to you.

Share via

Adding per-worker metrics reported to JMX in applicationinsights.json for App Insights

Your answer