This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 42%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
As the title says, I am receiving duplicate alert notifications when using Azure Managed Grafana. I believe that AMG uses high availability with two nodes, and this could be related to this issue that was opened in the Grafana GitHub, and that some users were able to resolve by ensuring proper network connectivity between the instances.
The aforementioned GitHub issue can be found here: https://github.com/grafana/grafana/issues/53990
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 43%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
@ADM-BEdwards - Thanks for reaching out to us.
I am currently investigating this issue and will get back to you as soon as possible.
Thank you for being patient while we work on this issue.
The current Grafana software does support alerting in high availability scenarios and they will synchronize with each other to prevent duplicate alerts as described in the GitHub issue I linked, more recently they updated their documentation to include the configuration requirements described in that GitHub issue. Unfortunately, because this is a managed offering, we as Azure administrators do not have the ability to update these files ourselves.
Here is the documentation: [https://grafana.com/docs/grafana/latest/alerting/set-up/configure-high-availability/#enable-alerting-high-availability
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 8%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYG%3C/text%3E%3C/svg%3E)
This works with Grafana running in Kubernetes but not on VMs, which is how our service hosts Grafana. We've been in active discussion w/ Grafana Labs but there is no resolution yet.
These instructions from the Grafana documentation that I linked apply to instances running on both VMs and on Kubernetes.
Enable alerting high availability in Grafana
Before you begin
Since gossiping of notifications and silences uses both TCP and UDP port 9094, ensure that each Grafana instance is able to accept incoming connections on these ports.
To enable high availability support:
[unified_alerting] section.[ha_peers] to the number of hosts for each Grafana instance in the cluster (using a format of host:port), for example, ha_peers=10.0.0.5:9094,10.0.0.6:9094,10.0.0.7:9094. You must have at least one (1) Grafana instance added to the [[ha_peer] section.[ha_listen_address] to the instance IP address using a format of host:port (or the Pod’s IP in the case of using Kubernetes). By default, it is set to listen to all interfaces (0.0.0.0).
We noticed your feedback that the above answer was not helpful. Thank you for taking time to share your feedback. Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback. Looking forward to your reply. Much appreciate your feedback.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 1%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWW%3C/text%3E%3C/svg%3E)
Hi @ADM-BEdwards , thanks for your feedback! We are considering how to improve the situation. Are you mainly complaining about that duplicated alerts are being sent to your alerting destination, or the duplicated alert state history in the Grafana UI is also not good? We want to understand which part is a more major concern. The alerting HA gossip protocol setup will not fix the duplicated alert history issue in the Grafana UI.
This is a limitation in the current Grafana software. Since we host Grafana on two VMs and there is no synchronization between them, you get duplicate alerts unfortunately. We've made a feature request to Grafana Labs. Hopefully they'll have a solution soon.
This HA implementation doesn't work for our current setup since it uses a static configuration. We use VMSS for hosting and rely on the Compute layer to assign IP addresses dynamically. We don't know what these addresses are ahead of the time to put into the configuration file. In addition, IP addresses can change when VMs need to be replaced. As mentioned, we're pushing for an alternative solution with Grafana Labs though we haven't gotten an agreement on that proposal yet.
ADM-BEdwards - Hope the above answer helped and please feel free to reach out if you have any further questions.
If the above response was helpful, please feel free to "Accept as Answer" and "Upvote" the same so it can be beneficial to the community.