Azure alerts terraform

gnuruwi 5

Hi, I'm working on set of alerts in azure to cover my Linux, Windows, SQLServers CPU, memory,disk, state etc. using terraform.

I have multiple account and in each i have multiple resource groups.

What should be the approcach to loop through all accounts, all resource groups.

What limitations i possibly could encounter.

BR,

AnuragSingh-MSFT 21,251 Reputation points

2023-10-26T07:20:23.93+00:00
gnuruwi, thank you for posting this question on Microsoft Q&A.

Based on my understanding, you have multiple Azure Subscriptions --> each containing multiple resource group --> and each resource group may contain one or more resources of same OR different types.

And you would like to update "all" existing resources of certain types (VMs, SQL Servers etc.) with some specific configuration (create alerts).

For the benefit of the community, I would like to add some basic details about Terrafrom which will be beneficial to understand the proposed solution. Before getting into details, note that Terraform follows a declarative pattern which means that the intended goals (or end results) are specified rather than the steps to reach that goal. The main component used for managing infrastructure using Terraform are - the configuration file (.tf) where the details of resource configuration are stored and the state file (.tfstate) which stores state about your managed infrastructure and configuration.

The information shared above is important, because if you want to update existing resources the following could happen:

If the existing resource was not created using terraform, the resources will first have to be imported into terraform state so that its engine understands the current state, before applying the next stage. This can be done using import command, but it needs some work - A sample scenario is discussed here - How to manage existing Azure Resource Groups using Terraform

If the existing resource was created using terraform and you have the state information in its backend, there are chances that updating existing resource might destroy or delete existing resource and create a new resource with the final desired configuration. The main reason for it is that Terraform takes an immutable approach to infrastructure, reducing the complexity of upgrading or modifying your services and infrastructure. Although there are ways to update existing resources without destroying them, there are a number of issues you would find online where update attempt causes resource deletion. You may also review the discussion here - Terraform and Updates

Here is the official doc for Azure resourc update using Terraform and a sample scenario - Change Azure infrastructure using Terraform. Note, it requires updates to existing resource file (.tf) to denote the target stare from current state of the resource.

Coming back to the primary question asked in this thread - updating the configuration of existing resource will require you to change the config file. I could not find specific tool/cmd to that, however, since the infrastructure code is stored as text file, you may use any scripting language (bash, PowerShell etc.) to loop through the .tf file and make the required changes in this file itself.

For your other part of the question - "... set of alerts in azure", I am not sure how are you planning to set these alerts at resource level. There are multiple ways to create alert rule depending on the trigger (metric based, log based, activity log based etc.). Infact, the alert rules are created as a separate resource with option to set the target resources, therefore they need not be created individually for each resource (VMs, SQL Servers etc.). Azure monitor is available as an independent provider with Terraform, and you can use the resource in it to achieve your end goal. Here are some links that should help:

azurerm_monitor_metric_alert

azurerm_monitor_scheduled_query_rules_alert

azurerm_monitor_activity_log_alert

Hope this helps. Please let me know if you have any questions.
gnuruwi 5 Reputation points

2023-10-26T08:31:34.9733333+00:00

Hi Anurag, Thanks for the detailed explanation.

Note: My current concentration is on to Get a alert from all VMs {Linux, Windows} on CPU,Memory,Disk and vm_state which spreaded across multiple accounts / multiple resource groups.

As of now i have created few alerts for CPU usage, Memory usgage, Disk utilization and VM state using 'azurerm_monitor_metric_alert'.

This was against 1 sbscription and 1 resources group and i have 2 VMs residing there.

Later i did recieved mail from Microsoft Azure azure-noreply@microsoft.com stating that Your Azure Monitor alert was triggered (CPU usage). Which means alert is working.

My next challenege i to get this to work across all resource groups in account ?

Is there any knows limitation to get this to work in terms of terraform ?

Thanks and looking forward to hear back.
AnuragSingh-MSFT 21,251 Reputation points

2023-10-26T09:50:55.2866667+00:00

gnuruwi, thank you for the reply. I will try to look into it and get back to this thread with what I find.

2 answers

AnuragSingh-MSFT 21,251 Reputation points

2023-10-30T07:58:15.6066667+00:00
gnuruwi, The "Metrics based alert rules" can be used to target all the resources in a single subscription, even though when residing in different resource groups in this subscription. Using azurerm_monitor_metric_alert provider, you may use the combination of the following arguments to achieve this:

Scope - Set this to Subscription resource id (for example - /subscriptions/<SUBSCRIPTION_ID>)

It seems that currently you have used Scope to target a single resource group, therefore the rule only gets applied to VM in that resource group. If the target is done at Subscription level, then all the resource groups inside this subscription and consequently all the VMs in these resource groups becomes the target.

Furthermore, criteria should be used to refine the target. For example, if the metric based alert rule is for VM, the criteria should specify the criteria.target_resource_type as Microsoft.Compute/virtualMachines

This will edit this metric rule to target all VMs in this specific scope (which the subscription in this case).

Regarding the limitations, the limitations as available for Subscription Management and Monitoring resources also applies when the resources are deployed through terraform:

Azure Subscription General limits

Azure Monitor service limits

Hope this helps. Please let me know if you have any questions.
Please sign in to rate this answer.

1 person found this answer helpful.
gnuruwi 5 Reputation points

2023-10-30T11:48:44.43+00:00

Thanks Anurag for much details.

As i said earlier i have few alerts defined in my main.tf for cpu, memort, disk and vm_state using "azurerm_monitor_metric_alert" and cpu alert is triggered and it's working.

Here's the alert criteria i used;

criteria { metric_namespace = "Microsoft.Compute/virtualMachines" metric_name = "Percentage CPU" aggregation = "Average" operator = "GreaterThan" threshold = 80 }

And i used similar criteria for memory, disk and vm_state, but terraform is throwing error indicating the used metric_name for memory, disk and vm_sate respectively.

metricalerts.MetricAlertsClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Couldn't find a metric named 'AvailableMemoryMB'. Make sure the name is correct.

criteria { metric_namespace = "Microsoft.Compute/virtualMachines" metric_name = "AvailableMemoryMB" aggregation = "Average" operator = "LessThan" threshold = 2048 }

Is there any recommended/approved metric_names to used for provider this case for Azure? From where i do find this information? Where/What shuld be the expected matric_name here for terraform Azure?

Best regards,

@ruwindada

AnuragSingh-MSFT 21,251 Reputation points

2023-11-02T09:36:39.68+00:00

gnuruwi, thank you for the reply.

Note that Azure provider for terraform uses Azure Resource Manager API for connecting with Azure. For more details, see Azure Provider

For references to Supported Metrics with Azure Monitor, see this link which contains information about the metric names for a particular resource in Azure's ARM API - Supported metrics with Azure Monitor

For Microsoft.Compute/virtualMachines, the metric names are available here - Supported metrics for Microsoft.Compute/virtualMachines

Note the column "Name in REST API", therefore the names of the AvailableMemoryMB will be:

AvailableMemoryMB - "Available Memory Bytes"

Hope this helps. Please let me know if you have any questions.

AnuragSingh-MSFT 21,251 Reputation points

2023-11-03T07:10:00.5633333+00:00

gnuruwi, Following-up to check if you had a chance to review the answer provided.

If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

gnuruwi 5 Reputation points

2023-11-05T07:10:21.7633333+00:00

Thanks for the reply Anurag it helped me to get the metric_names correctly and my alerts are working fine!

AnuragSingh-MSFT 21,251 Reputation points

2023-11-06T05:03:49.58+00:00

gnuruwi, thank you very much for the confirmation. I am glad that I was able to help you with this issue.

Please click Accept answer so that it can help others in the community looking for help on similar topics.
Sign in to comment
gnuruwi 5 Reputation points

2024-05-07T07:39:53.84+00:00

Hi Anurag, Coming back to topic and i found this link all most all the Metric alerts and related properties.

https://azure.github.io/azure-monitor-baseline-alerts/welcome/

And further wanted ask question about resource types which do not support "Multi-resource alerts" as of now.

For example, i need to configure set of alerts for Storge account and Load Balancer and both do not support for multi-resource alerts. I have requested these by voting in the portal. Any idea about this feature availability?

At the moment, only way to get this done is by specifying under laying resource group of storage account or the load balancer. Do you see better approach for this ?

Thanks in advance.

@Ruwinda Fernando
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Azure alerts terraform

2 answers