Eventgrid of machine learning work space not receiving events

Question

Eventgrid of machine learning work space not receiving events

Elias Asikainen 50

Machine learning workspace run status changed events are no longer being sent to event grid

Ricardo Elizondo 35 Reputation points

2026-03-02T11:35:05.9466667+00:00

Microsoft.MachineLearningServices.RunStatusChanged currently fires only when a run transitions into a Failed state.

This is not true, at least not up Feb27, 2026 (when our event trigger stopped working). This was triggered for any change of status, be it failed, canceled, completed, etc.

The documentation also simply mentions 'raised when a run status changes. Could you reference documentation that specifies this behavior?
Elias Asikainen 50 Reputation points

2026-03-02T12:06:32.1466667+00:00

As Ricardo describes it. All status changes triggered the events before it stopped working for us as well on Feb 27.
Alejandro Oliva 10 Reputation points

2026-03-02T14:14:50.5166667+00:00

Same issue here. Our workspace in West Europe stopped publishing RunStatusChanged events to Event Grid on February 27, after 13:00 UTC.

Published Events metric on the System Topic shows 0 since then. We also tried switching to RunCompleted and it doesn't work either.

Jobs complete successfully, Microsoft.EventGrid provider is registered, System Topic points to the correct workspace. No changes were made on our side. This is blocking our production pipeline.
Ciprian Vicentiu Dragusin Vasile 5 Reputation points

2026-03-02T14:58:51.4666667+00:00

We are experiencing the same behaviour. the last incoming events were on 27th of February 2026. We have even recreated the Event Grid System Topic and hooked the Azure Function triggers from the incoming Azure Machine Learning events and no event at all is getting registered in the Sytem Topic blade interface. This is generalised over all of our System Topics incoming from Machine Learning and we haven't changed a thing. Please investigate this issue in depth because it's coming form the Azure side and all of our productive processes are failing.
Elias Asikainen 50 Reputation points

2026-03-02T15:05:18.7066667+00:00

I have also created a support request for this issue and currently the ML team is looking into it. Will update this post if I get more info. We also made no changes and the events just stopped.
Eric Giddings 1 Reputation point

2026-03-02T16:25:15.4366667+00:00

I am also experiencing this same issue with multiple workspace and Event Grid implementations. It stopped suddenly with no changes. It has been since possibly 2/25. The issue started around the same time as everyone else in here is saying. None of the Moderator's solutions have worked. Seems like an unreported Azure outage.
Elias Asikainen 50 Reputation points

2026-03-02T19:10:38.1966667+00:00

I got an answer:
"We want to inform you that there is an ongoing issue in the West Europe region where AML has not been sending events to the event grid since February 27th.

We'll let you know as soon as we have more details."

Will keep you posted.
Ricardo Elizondo 35 Reputation points

2026-03-03T09:51:56.7966667+00:00

Update on our side: issue resolved itself on all our environments around 8:30AM UTC+01:00 March 3, for our Event Grid resources in West Europe.
Sina Salam 27,971 Reputation points Volunteer Moderator

2026-03-03T14:13:09.9033333+00:00

Hello Elias Asikainen,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your Eventgrid of machine learning work space not receiving events.

To resolve the issue, ensure your Azure Machine Learning workspace Event Grid subscription includes both Microsoft.MachineLearningServices.RunCompleted and Microsoft.MachineLearningServices.RunStatusChanged events as defined in the official event schema - https://learn.microsoft.com/azure/event-grid/event-schema-machine-learning, verify the subscription state, filters, and metrics in the Azure portal, test with a forced failed run to confirm event emission, enable diagnostic logs and delivery metrics per Event Grid troubleshooting guidance - https://learn.microsoft.com/azure/event-grid/troubleshoot-issues, validate endpoint health and authentication (HTTP 200–204 required), and configure dead-lettering following delivery and retry policies - https://learn.microsoft.com/azure/event-grid/delivery-and-retry to conclusively determine whether the issue lies in event publication, filtering, or endpoint delivery.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
SRILAKSHMI C 14,820 Reputation points Microsoft External Staff Moderator

2026-03-05T07:18:37.9333333+00:00

Hi Elias Asikainen,

Could you please confirm whether the issue is still persisting on your end, or if it has already been resolved?

Thank you!
Elias Asikainen 50 Reputation points

2026-03-05T07:19:58.8233333+00:00

Issue is also resolved on our side. Thanks
SRILAKSHMI C 14,820 Reputation points Microsoft External Staff Moderator

2026-03-06T05:47:38.02+00:00

Hi Elias Asikainen,

Thank you for the confirmation!
Ricardo Elizondo 35 Reputation points

2026-03-06T09:24:44.5533333+00:00

@SRILAKSHMI C I can't find the outage reported by Azure anywhere, is there one?
The service was basically down from Feb 27th to March 3rd, so there must be an outage report, given this is way past the SLA for both Event Grid System Topics and Azure Machine Learning Workspace.
Checked both Azure Service Health and the Azure Status website.

Thanks!

1 answer

Your answer

Ricardo Elizondo 35 Reputation points

2026-03-02T11:35:05.9466667+00:00

Microsoft.MachineLearningServices.RunStatusChanged currently fires only when a run transitions into a Failed state.

This is not true, at least not up Feb27, 2026 (when our event trigger stopped working). This was triggered for any change of status, be it failed, canceled, completed, etc.

The documentation also simply mentions 'raised when a run status changes. Could you reference documentation that specifies this behavior?
Elias Asikainen 50 Reputation points

2026-03-02T12:06:32.1466667+00:00

As Ricardo describes it. All status changes triggered the events before it stopped working for us as well on Feb 27.
Alejandro Oliva 10 Reputation points

2026-03-02T14:14:50.5166667+00:00

Same issue here. Our workspace in West Europe stopped publishing RunStatusChanged events to Event Grid on February 27, after 13:00 UTC.

Published Events metric on the System Topic shows 0 since then. We also tried switching to RunCompleted and it doesn't work either.

Jobs complete successfully, Microsoft.EventGrid provider is registered, System Topic points to the correct workspace. No changes were made on our side. This is blocking our production pipeline.
Ciprian Vicentiu Dragusin Vasile 5 Reputation points

2026-03-02T14:58:51.4666667+00:00

We are experiencing the same behaviour. the last incoming events were on 27th of February 2026. We have even recreated the Event Grid System Topic and hooked the Azure Function triggers from the incoming Azure Machine Learning events and no event at all is getting registered in the Sytem Topic blade interface. This is generalised over all of our System Topics incoming from Machine Learning and we haven't changed a thing. Please investigate this issue in depth because it's coming form the Azure side and all of our productive processes are failing.
Elias Asikainen 50 Reputation points

2026-03-02T15:05:18.7066667+00:00

I have also created a support request for this issue and currently the ML team is looking into it. Will update this post if I get more info. We also made no changes and the events just stopped.
Eric Giddings 1 Reputation point

2026-03-02T16:25:15.4366667+00:00

I am also experiencing this same issue with multiple workspace and Event Grid implementations. It stopped suddenly with no changes. It has been since possibly 2/25. The issue started around the same time as everyone else in here is saying. None of the Moderator's solutions have worked. Seems like an unreported Azure outage.
Elias Asikainen 50 Reputation points

2026-03-02T19:10:38.1966667+00:00

I got an answer:
"We want to inform you that there is an ongoing issue in the West Europe region where AML has not been sending events to the event grid since February 27th.

We'll let you know as soon as we have more details."

Will keep you posted.
Ricardo Elizondo 35 Reputation points

2026-03-03T09:51:56.7966667+00:00

Update on our side: issue resolved itself on all our environments around 8:30AM UTC+01:00 March 3, for our Event Grid resources in West Europe.
Sina Salam 27,971 Reputation points Volunteer Moderator

2026-03-03T14:13:09.9033333+00:00

Hello Elias Asikainen,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your Eventgrid of machine learning work space not receiving events.

To resolve the issue, ensure your Azure Machine Learning workspace Event Grid subscription includes both Microsoft.MachineLearningServices.RunCompleted and Microsoft.MachineLearningServices.RunStatusChanged events as defined in the official event schema - https://learn.microsoft.com/azure/event-grid/event-schema-machine-learning, verify the subscription state, filters, and metrics in the Azure portal, test with a forced failed run to confirm event emission, enable diagnostic logs and delivery metrics per Event Grid troubleshooting guidance - https://learn.microsoft.com/azure/event-grid/troubleshoot-issues, validate endpoint health and authentication (HTTP 200–204 required), and configure dead-lettering following delivery and retry policies - https://learn.microsoft.com/azure/event-grid/delivery-and-retry to conclusively determine whether the issue lies in event publication, filtering, or endpoint delivery.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
SRILAKSHMI C 14,820 Reputation points Microsoft External Staff Moderator

2026-03-05T07:18:37.9333333+00:00

Hi Elias Asikainen,

Could you please confirm whether the issue is still persisting on your end, or if it has already been resolved?

Thank you!
Elias Asikainen 50 Reputation points

2026-03-05T07:19:58.8233333+00:00

Issue is also resolved on our side. Thanks
SRILAKSHMI C 14,820 Reputation points Microsoft External Staff Moderator

2026-03-06T05:47:38.02+00:00

Hi Elias Asikainen,

Thank you for the confirmation!
Ricardo Elizondo 35 Reputation points

2026-03-06T09:24:44.5533333+00:00

@SRILAKSHMI C I can't find the outage reported by Azure anywhere, is there one?
The service was basically down from Feb 27th to March 3rd, so there must be an outage report, given this is way past the SLA for both Event Grid System Topics and Azure Machine Learning Workspace.
Checked both Azure Service Health and the Azure Status website.

Thanks!

Answer 1

Hello Elias Asikainen,

Welcome to Microsoft Q&A and Thank you for reaching out.

If Machine Learning workspace run status changed events are no longer being delivered to Event Grid, and this was previously working, the issue is typically related to event type expectations, subscription configuration, endpoint health, or identity/network changes.

Steps to troubleshoot the issue,

1.Understand What RunStatusChanged Actually Emits

Microsoft.MachineLearningServices.RunStatusChanged currently fires only when a run transitions into a Failed state.

If your runs are completing successfully or being cancelled normally, you will not receive this event.

If you need notification for successful completion as well, make sure you also subscribe to:

Microsoft.MachineLearningServices.RunCompleted

This is a very common misunderstanding and often the root cause.

2.Verify the Event Subscription Configuration

Go to Azure Portal → Machine Learning Workspace → Events → Event Subscriptions

Check The subscription status is Succeeded

The correct event types are selected:

RunStatusChanged
(Optional but recommended) RunCompleted

No advanced filters are unintentionally filtering events

The subscription hasn’t expired (webhook-based subscriptions can expire)

Also review Delivery Status

Metrics blade

Published Events
Matched Events
Failed Deliveries

If Published Events = 0, either Events are not being emitted Or filters are excluding them

3.Validate Endpoint Health & Networking

If your subscription delivers to:

Webhook

Azure Function

Logic App

Event Hub

Service Bus

Verify Endpoint is reachable

No recent networking changes (VNET, private endpoint, firewall rules)

No authentication changes (key rotation, managed identity change)

Endpoint isn’t returning 401/403/500 errors

In Event Grid Check Delivery Logs

Confirm validation handshake is successful

If validation status shows red, recreate or revalidate the subscription

Event Grid retries for a limited time before dropping events.

4.Test with a Failing Run

To confirm the pipeline:

Trigger a test run that you intentionally force to fail.

Monitor Event Grid metrics in real time.

If you receive RunStatusChanged, then:

The event system is working correctly.

The issue was expectation-related (only fires on failure).

If you still receive nothing the issue is likely subscription-level or workspace-level.

5.Confirm Events Are Being Published

Go to Azure Portal → Event Grid → System Topics

Check Is there a system topic associated with your ML workspace?

Are Published Events increasing?

If Published Events = 0 even for failing runs:

The workspace may have been recreated (new resource ID)

The event subscription may be pointing to an old resource

There may be a service-side issue (check Azure Service Health)

If the workspace was deleted and recreated (even with same name), the old event subscription will stop working.

6.Identity / RBAC Changes

If using Managed Identity, SAS token, AAD authentication

Verify Required RBAC roles still exist, Managed identity wasn’t recreated

Keys weren’t rotated

No permission changes occurred

Silent RBAC changes are a very common cause of event delivery failures.

7.Enable Diagnostics for Deeper Insight

Enable diagnostic logs on the Event Grid topic:

Send logs to Log Analytics or Storage Account

Review delivery failures and retry attempts

You can also use:

az eventgrid event-subscription show --name <subscription-name> --source-resource-id <workspace-resource-id>

To inspect configuration and delivery settings.

Isolation Test

Create a new temporary event subscription:

Point it to a simple Azure Function or Event Hub

Subscribe to both:

RunStatusChanged
RunCompleted

Trigger a failing run

If new subscription works → issue is endpoint-related If not → issue is workspace/system topic-related

Common Root Causes

In real-world cases, the issue is usually:

Expecting RunStatusChanged for successful runs
Endpoint authentication change
Workspace recreated (new resource ID)
Event subscription expired
Networking changes (private endpoints/firewalls)
RBAC changes

Please refer this

https://learn.microsoft.com/azure/event-grid/event-schema-machine-learning

https://learn.microsoft.com/azure/machine-learning/how-to-use-event-grid#consume-machine-learning-events

Troubleshooting common Event Grid issues - https://learn.microsoft.com/azure/event-grid/troubleshoot-errors

I Hope this helps. Do let me know if you have any further queries.

Thank you!

Share via

Eventgrid of machine learning work space not receiving events

1 answer

Your answer