cross tenant alerting

Question

Hi,following on from this notification

https://azure.microsoft.com/en-us/updates/sending-a-log-search-alert-with-cross-tenant-target-resource-will-no-longer-be-supported/

the part 'As of March 15, 2024, this behavior will change and sending a log search alert with a cross tenant target resource (except for the lighthouse case) will no longer be supported.'

I use azure lighthouse to manage a number of customer subscriptions, up until the above date I used an Azure monitor log search alert to query all log anayltic workspaces in customer delegated subscriptions.

and as I use lighthouse I (wrongly) assumed that 'except for the lighthouse case' would mean that lighthouse behaviour would not change, until all the alerts stopped working.

the alert query rule is saved in the managing tenant, when it has a condition to alert on , it returns an error 'Alert is blocked due to unauthorized target resource (target resource tenant is different from rule tenant).'

that i thought was sort of the point of lighthouse, create a rule in the managing tenant so that you do not need monitor rules in every customer for the same thing. ie. rather than have x number of low disk space alerts , you just need 1 in the managing tenant that will create alerts when the conditions are met.

the KQL works from log analytics so it doesnt appear to be an issue reading the data from the customer workspace.

this was used a basis for the intial alert setup

https://learn.microsoft.com/en-us/azure/lighthouse/how-to/monitor-at-scale#query-data-across-customer-workspaces

this document however hasnt been updated after the cross tenant bulletin at the top of this question so i am not sure exactly how a cross tenant log search alert will work now with lighthouse if we cannot have the alert rule saved in the managing tenant.

I havent found any documention that explains how cross tenant alerting will now work after the march 15th changes.

anyone else had a similar situation that can maybe shine a light on this ?

many thanks

Answer 1

just to update this question.

I've had a case open for a while now regarding this and much debugging has been done to try and understand why the cross tenant alerts were not working.

however, on 30th July all our alerts starting working again without any changes by us or by the support team dealing with our request.

so thats a piece of good news, hopefully they have started worked for anyone else affected.

there is a request for a root cause for this to understand what happened but as of now it is not clear why they broke and why they started working again months later.

Answer 2

I have now got the description of the root cause of this error from Microsoft support and it appears it was completely an issue on their side and not our alert configuration.

hopefully anyone else affected by this now has alerts working again.

Root cause provided was

'I was able to get some further confirmation on the exact issue we faced with Azure alerts not working in conjunction with Azure Lighthouse.  

I spoke with both teams and it seems this issue was completely on our side. Meaning, we had the Lighthouse implementation issue with the access it has user subscriptions.  

More specifically it didn't add read permissions to our first party application, we just changed the scripting these permission assignments rely on. We also improved the logic to differentiate between situation we don't have the permissions and situations in which the user/rule doesn't have the permission.  

Specific impact range for the incident(this was also raised separately from this request as well) matches with what we see as successful triggering on your end.  

Once this was fixed, alerts could trigger successfully once more. '