Inconsistent, intermittent errors on deployed app, cannot be replicated locally

Question

Inconsistent, intermittent errors on deployed app, cannot be replicated locally

Ellen 0

We recently implemented blue-green deployment and now when we run the system after deploying an update it produces CORS errors only sometimes, and some of the deployed changes sometimes appear to not be implemented. We do have multiple instances, but autoscaling is off. This seems to be related to which instance the code is running on.

Sai Prabhu Naveen Parimi 2,260 Reputation points Microsoft External Staff Moderator

2025-06-19T18:01:17.1966667+00:00

@Ellen

To assist further with our investigation, could you please confirm which Azure service area your application is deployed in ?
Siva Nair 2,340 Reputation points Microsoft External Staff Moderator

2025-06-20T19:14:40.36+00:00
Hi Ellen,

These can happen for several reasons, especially when different instances of your app are trying to handle requests. Make sure your CORS policy is correctly configured to allow requests from all necessary domains. You want to ensure that any domains you’re making requests from are included in your CORS settings.

This deployment strategy can sometimes lead to issues if not all instances are running the same version of your application. Verify that all instances are indeed running the latest version of your code. As you mentioned, autoscaling is off, so inconsistency might arise if there are leftover instances still serving traffic from previous deployments.

Since you've noted that it behaves differently based on the instance, use Application Insights to monitor the health of each instance during and after deployment. Look for differences in error rates or logs that might indicate which instance is misbehaving and why.

Sometimes, browsers or servers cache responses, leading to the perception of changes not being applied. Make sure to clear caches as needed, or implement cache-busting techniques where applicable.

Quick action list you can try:

Check and update your CORS policy to ensure all required origins are allowed.

Inspect each instance’s deployment and make certain they are all up-to-date.

Utilize Application Insights to track down where the errors are occurring.

Clear caches in browsers and possibly on your servers if you're using CDN layers.

To assist further, could you provide a bit more detail on the following?

Which Azure service are you using to deploy your application?

Are there any specific error messages appearing alongside the intermittent CORS issues?

Have you implemented specific logging or monitoring solutions for your instances?

Do you have a readily accessible list of the URLs or domains that need to be CORS-enabled?

Thanks!
Siva Nair 2,340 Reputation points Microsoft External Staff Moderator

2025-06-23T17:55:22.55+00:00

Hi Ellen,

Just checking back if the above response was helpful, if you have any questions or concerns please, feel free to post back.

Moreover, to troubleshoot further please provide the detailed asked in above response.

Thanks
Venetta Smithen 80 Reputation points

2025-06-23T18:11:29.1666667+00:00
This issue possibly points to imcomplete traffic switching, caching/session issues or environment mismatches between your blue and green deployments.

In blue-green deployment, traffic should be fully routed to one environment (either blue or green). If it's split or sticky sessions are involved, you may hit old instances. Symptoms of CORS errors or missing updates can be noted depending on which environment is hit.

This can occur due to load balancer or DNS switching not being switched for all requests to the new environment.

Check to:

Ensure proper Cache-control, ETag and versioning strategies on static assets.

Review backend CORS settings on all active instances

Disable stickiness or purge old environments

Fixes: Verify Traffic Cutover: Ensure load balancer or DNS fully switches traffic to the new (green) environment before tearing down old one.

Confirm there is no routing rules pointing to the old (blue) environment after the switch.

Audit CORS config

Make sure all instances (green deployment) have identical CORS settings.

Check your backend headers for:

Access Control Allow Origin

Access Control Allow Methods

Access Control Allow Headers

Clear Cache

Set appropriate cache headers to avoid client-side caching:

http

Cache-Control: no cache. no-store, must-revalidate

Consider adding asset versioning to JS/CSS/HTML files

Remove old instances - fully decommission the old (blue) environment

Test

Use curl, browser DevTools or Postman to simulate requests from various origins and inspect which instance is responding.

Your answer

Sai Prabhu Naveen Parimi 2,260 Reputation points Microsoft External Staff Moderator

2025-06-19T18:01:17.1966667+00:00

@Ellen

To assist further with our investigation, could you please confirm which Azure service area your application is deployed in ?
Siva Nair 2,340 Reputation points Microsoft External Staff Moderator

2025-06-23T17:55:22.55+00:00

Hi Ellen,

Just checking back if the above response was helpful, if you have any questions or concerns please, feel free to post back.

Moreover, to troubleshoot further please provide the detailed asked in above response.

Thanks
Venetta Smithen 80 Reputation points

2025-06-23T18:11:29.1666667+00:00

This issue possibly points to imcomplete traffic switching, caching/session issues or environment mismatches between your blue and green deployments.

In blue-green deployment, traffic should be fully routed to one environment (either blue or green). If it's split or sticky sessions are involved, you may hit old instances. Symptoms of CORS errors or missing updates can be noted depending on which environment is hit.

This can occur due to load balancer or DNS switching not being switched for all requests to the new environment.

Check to:

Ensure proper Cache-control, ETag and versioning strategies on static assets.

Review backend CORS settings on all active instances

Disable stickiness or purge old environments

Fixes: Verify Traffic Cutover: Ensure load balancer or DNS fully switches traffic to the new (green) environment before tearing down old one.

Confirm there is no routing rules pointing to the old (blue) environment after the switch.

Audit CORS config

Make sure all instances (green deployment) have identical CORS settings.

Check your backend headers for:

Access Control Allow Origin

Access Control Allow Methods

Access Control Allow Headers

Clear Cache

Set appropriate cache headers to avoid client-side caching:

http

Cache-Control: no cache. no-store, must-revalidate

Consider adding asset versioning to JS/CSS/HTML files

Remove old instances - fully decommission the old (blue) environment

Test

Use curl, browser DevTools or Postman to simulate requests from various origins and inspect which instance is responding.

Share via

Inconsistent, intermittent errors on deployed app, cannot be replicated locally

Your answer