Azure AI Foundry Project Prompt Flow deployments fail for any new project or Hub

Question

For the past week Prompt Flow deployments under AI Foundry Projects were failing for any new AI Foundry HUB or Project. Existing old projects and online endpoints work fine, but in any newly created HUB or project when deploying Prompt Flow, it never executes and deploys it successfully. Deployment might run over 30-40 minutes fail and retry and fail after another 30 minutes to 1hour.

Errors we get are the following:

This one comes after second failure in the ARM logs

2025-06-09T20:35:53Z Check envoy cert setting failed in MirSystemSetupTask. Please check the existence/validation of envoy cert.

This one comes after first failure

Conflict Status: 409 (Conflict) ErrorCode: Conflict Content: { "error": { "code": "Conflict", "message": "Conflict", "details": [ { "code": "InferencingClientCallFailed", "message": "\"Request could not be completed due to a conflict with the current state of the target resource, Please try again later. Already running method StartCreateDeploymentAsync with operation [a3832ec0-8111-4db0-869f-af67993a6511]. Can not perform StartUpdateDeploymentAsync.\"", "details": [], "additionalInfo": [] } ], "additionalInfo": [ { "type": "ComponentName", "info": { "value": "managementfrontend" } }, { "type": "Correlation", "info": { "value": { "operation": "901d37c8622f4a7b80139d937b9d338f", "request": "7ea73541cd9fa41b" } } }, { "type": "Environment", "info": { "value": "westeurope" } }, { "type": "Location", "info": { "value": "westeurope" } }, { "type": "Time", "info": { "value": "2025-06-11T11:03:32.7120707+00:00" } } ] } } Headers: Cache-Control: no-cache Pragma: no-cache x-ms-operation-identifier: REDACTED Request-Context: REDACTED x-ms-response-type: REDACTED Strict-Transport-Security: REDACTED X-Content-Type-Options: REDACTED azureml-served-by-cluster: REDACTED x-request-time: REDACTED x-ms-throttling-version: REDACTED x-ms-ratelimit-remaining-subscription-resource-requests: REDACTED x-ms-request-id: 8132bebc-2b6b-4840-8245-d342b7247551 x-ms-correlation-request-id: REDACTED x-ms-routing-request-id: REDACTED X-Cache: REDACTED X-MSEdge-Ref: REDACTED Date: Wed, 11 Jun 2025 11:03:32 GMT Content-Length: 1201 Content-Type: application/json; charset=utf-8 Expires: -1

For existing projects we can successfully create new Prompt lLows and deploy them to new or existing pre-created online endpoints. But even under the same HUB if we create a new project and try Prompt Flow deployment to new or existing pre-created online endpoint, we constantly get these errors that are not descriptive at all.

Were there some underlying changes to the service or some role assignment that should be done but not propagated that are affecting the service? This has been tested in multiple subscription under multiple tenants in various regions (Sweden Central, West Europe and North Europe) and the behavior is exactly the same everywhere.