A server in Azure is taking a long time to complete a task

Question

Hi, we are are experiencing an odd issue, where we have a letter generating application that runs on Azure server A for Test and Live code that communicates with Server B and Server C. All our test letter templates are saved on server B and when the application on server A runs the test code using letter templates from server B, it takes 10 -20 seconds to generate a letter. But when the live code runs using the letter templates from server C, it takes 2 min which is a very long time. We have recently migrated all these 3 servers from on-prem to Azure and they have been running fine when they were on-prem. Just to mention, all these 3 servers are in the same vlan and the only difference is server A and B are in one subnet and sever C is in a different subnet. We have premium disks and high performance tier for our disks on servers and analyzed some performance metrics that don't suggest the issue to be related to the resource allocation.

We have also checked the code sitting on server A for both test and live but cant see any difference. We would appreciate if we can get some advice / suggestion on this as do we need to make any firewall changes or any other advice that could point us to the right direction, that would be much appreciated. Many thanks

Answer 1

Hi @Ghulam Abbas

Thank you for reaching out to the Microsoft Q&A platform.

The performance difference between your test and live environments, with both running on Azure, can be challenging to diagnose. It's great that you've ruled out resource allocation issues and checked the code for differences. However, there are several other factors you should consider pinpointing the performance bottleneck:

Azure Region: Make sure that both Server B and Server C are in the same Azure region as Server A. Latency between regions can impact performance.

Network and Subnet: You've mentioned that Server A and Server B are in one subnet, while Server C is in a different subnet. Ensure that there's no unnecessary network traffic between these subnets that could cause latency. Check for Network Security Groups (NSG) and route tables that might be affecting communication.

Azure Disk Performance: You mentioned using premium disks with high-performance tiers, but you may also want to ensure that your disk performance is not bottlenecked, especially on Server C. Monitor disk I/O and latency metrics to see if any issues are evident.

Firewall Rules: Review the firewall rules and any security configurations on Server C. There might be rules or configurations that are causing additional delays for Server C compared to Server B.

Azure Diagnostics and Monitoring: Azure provides extensive diagnostic and monitoring tools. Utilize Azure Monitor and Application Insights to track performance issues and identify bottlenecks. Set up alerts to get notified of anomalies.

Database and Data Access: If your application communicates with a database, ensure the database on Server C is configured correctly and optimized for the live environment. Database performance can significantly impact application speed.

Code Profiling: Use code profiling tools to identify any bottlenecks in the live code that might not be present in the test code. Pay attention to areas that involve data access, external API calls, or complex calculations.

Logging and Error Handling: Analyze the logs and error messages in both the test and live environments to see if there are any errors or issues specific to the live environment that might be causing delays.

Third-Party Services: Check if any third-party services are being used in the live environment that could be causing delays, such as external APIs or authentication services.

If you are unable to identify the issue after thorough analysis, consider reaching out to Azure support. They can provide more specific guidance and assistance in diagnosing performance problems.

Remember to make changes cautiously and always have a rollback plan in place, especially in a live environment. It's important to isolate and test changes in a controlled environment before applying them to your live system to avoid any potential issues.

Hope this helps!

Answer 2

Many thanks