Active worker nodes usage in HDinsight cluster for the period of last one year

Question

Hi Team,

Can someone help us to get the Active worker nodes usage in HDinsight cluster for the period of last one year.

Thanks & Regards,
Udaya

Answer 1

Hello and welcome to Microsoft Q&A @UDAYA SRINIVASARAO KOTHAMASU .

If I understand correctly, you want to get historical data for active nodes. With the appropriate preparations this would be easy to do.

Have you configured Azure Monitor to save the logs from your HDInsight cluster, or set up the Diagnostic Settings to save your logs? If so, this should be easy to do. If not, I am uncertain whether the entire year's data can be retrieved. Generally, there is a limit to historical data.

Some data can be retrieved from the portal without setting up Diagnostic Settings or Azure Monitor:

Under Monitoring: Metrics, the number of active worker nodes is available. This data extends to the past 30 days.

When autoscale is enabled, the scale-up, scale-down is recorded in the activity log. This records the number of nodes. The activity logs can be downloaded/exported.