Azure VM Server Comunication problem with On premises computers

Question

Azure VM Server Comunication problem with On premises computers

Dalbir Singh /IT /DB 0

Dear All

I hope this message finds you well. I am writing to bring to your attention a critical issue we have been encountering in our communication with Azure VM servers and local on-premise computers, adversely affecting our day-to-day operations.

During office hours, User attempts to upload or download files, even of moderate sizes such as a 150 MB PDF, result in significantly extended copy times, often exceeding the usual duration of 1 to 2 hours. Furthermore, these operations intermittently become unresponsive, causing disruptions in our workflow.

Our Azure VM server, where users upload files, is accessed through a firewall with a public IP configured in the gateway, adhering to the recommended configuration. However, we have observed an irregular ping status between the firewall and the Azure VM server, characterized by numerous drops and occasional latency spikes.

Despite engaging our lease line service provider and initiating a thorough investigation, no irregularities were found from their end. Local network operations remain unaffected, with users experiencing normal communication and data transfer speeds.

We escalated our concerns to the Microsoft support team, yet no issues were identified on their part. The problem persists, significantly impacting our efficiency.

We seek your expertise to identify any potential oversights from our end that may be contributing to this issue. Your timely and valuable suggestions are crucial in resolving this matter promptly. We appreciate your attention to this urgent matter and look forward to your guidance.

Thanks & Regards

Dana Akary 0 Reputation points

2023-11-14T09:08:16.1333333+00:00

Hi

Things to consider

Azure VM Location:

Azure provides data centers in various regions around the world. When you create a VM, you choose the Azure region where the VM will be deployed. Selecting a region closer to your physical location can reduce network latency and potentially improve performance. Here you can find details about the azure regions https://azure.microsoft.com/en-gb/explore/global-infrastructure/geographies/#choose-your-region

Azure VM Sizes and Network Performance:

Azure provides a variety of Virtual Machine (VM) sizes, and each size comes with different specifications, including CPU, memory, and network bandwidth. When you create a VM, you choose a specific size based on your application's requirements. It's crucial to select a VM size that offers sufficient network bandwidth for your workload. You can read about the network bandwidth of the different VM sizes here https://learn.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series

VM Disk Performance:

Slow upload speeds could be related to disk I/O. If you are uploading or downloading data to/from disks on the VM, ensure that the disk configuration (example: Standard HDD, Standard SSD, Premium SSD) meets your performance requirements. Check this link for more information about disk sizes and performance https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types

Network Monitoring:

Use Azure Monitor to monitor network performance and identify any bottlenecks or issues. Look for network metrics in the Azure Portal or use Azure Monitor logs.

Network Security Group Rules:

Review your Network Security Group (NSG) rules. Ensure that you are not restricting the necessary traffic and that the rules are appropriately configured.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-14T09:28:04.84+00:00
@Dalbir Singh /IT /DB

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

May I ask how the Azure VM and OnPrem users are connected?

Is it via Internet ?

Are you uploading
from Azure VM to OnPrem via Internet?
or from Azure VM to an Internet Site?
or from OnPrem server to Azure VM via Internet?

May I ask where the firewall comes in?

Is it placed in Azure or in OnPrem

Can you please share the network flow on the upload.

Please note that in cases where traffic is uploaded via Internet, no bandwidth can be guaranteed as Azure has little control over the underlying Internet infrastructure.

Thanks,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-15T05:24:58.6666667+00:00

Dear All

Thanks to everyone for the valuable suggestions.

Our setup is in North India, and the Azure region selected is in Central India.

Azure VM and OnPrem users connected with the internet.

Upload and download flow is via the internet. The connection was established with the WAN IP configured in the firewall.

Firewall is configured on Prem; only VMs are configured in Azure Portal.

Earlier, it was working fine, but since the last two months, we have been getting drops while connecting to On Prem systems with Azure VM.

Do we require any changes in our Azure tunnel or firewall! Something is missing from our end?
Dana Akary 0 Reputation points

2023-11-16T07:59:29.94+00:00

If you connect to the VM from outside the company network do you still experience the same?
Dalbir Singh /IT /DB 0 Reputation points

2023-11-16T08:49:39.8166667+00:00

While connecting with the VPN client from different internet connection, file copy takes time; the 340 MB file size shows 1 hour, 30 minutes of time remaining.
Dana Akary 0 Reputation points

2023-11-16T09:18:14.9933333+00:00

Another test you can try to further isolate the issue, If you have a different VM in the same network try to upload a file with the same size to the affected VM.

Monitor as well the CPU and Disk performance of the VM itself during the copy process.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-16T09:37:09.2+00:00

@Dalbir Singh /IT /DB

Thanks for sharing your environment.

Since the connection takes place via Internet, we cannot comment much on the latency or bandwidth (thus impacting the performance).

As stated by Dana Akary, you can try to reproduce this issue from a similar VM in the same VNET as this VM and isolate if this is specific to a VM or to the entire VNET.

You can try using SFTP or FTP Clients such as WinScp and check if the performance improves.

Cheers,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-17T05:02:12.2966667+00:00

Thanks everyone for your valuable time & Suggestion.

One strange thing we noticed in this activity. while pinging our Azure VM Server Public IP, No such drops reported. Alongside while pinging our local private IP we are getting drops & latency. Somewhere around in 3 digit.

Reply from 10.*** * *: bytes=32 time=263ms TTL=128

Reply from 10.*** * *: bytes=32 time=151ms TTL=128

Reply from 10.*** * *: bytes=32 time=414ms TTL=128

Reply from 10.*** * *: bytes=32 time=144ms TTL=128

Do we need any changes, Please let us know .........

Thanks again.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-20T11:42:28.69+00:00
@Dalbir Singh /IT /DB

There is an ongoing issue on latency of Azure VPN gateway : https://learn.microsoft.com/en-us/answers/questions/1383683/azure-vpn-latency-every-8-12-seconds.

We have created a document item to call out the latency issue :

https://github.com/MicrosoftDocs/azure-docs/issues/116873

To troubleshoot latency issues, We will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue. If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-21T06:33:00.8366667+00:00

@KapilAnanth-MSFT

Thanks for the update.

We are working on the issue with the Azure support engineer. Hopefully, we will resolve the issue timely.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-21T11:40:37.38+00:00

@Dalbir Singh /IT /DB

Thanks for the update.

Hopefully, the issue gets resolved soon.

Cheers,

Kapil.
Dana Akary 0 Reputation points

2023-11-22T11:36:58.8133333+00:00

Please try the steps explained here https://learn.microsoft.com/en-us/azure/vpn-gateway/reset-gateway to reset the VPN Gateway? Lets see the ping results and upload speed after the reset.

Dana Akary 0 Reputation points

2023-11-14T09:08:16.1333333+00:00

Hi

Things to consider

Azure VM Location:

Azure provides data centers in various regions around the world. When you create a VM, you choose the Azure region where the VM will be deployed. Selecting a region closer to your physical location can reduce network latency and potentially improve performance. Here you can find details about the azure regions https://azure.microsoft.com/en-gb/explore/global-infrastructure/geographies/#choose-your-region

Azure VM Sizes and Network Performance:

Azure provides a variety of Virtual Machine (VM) sizes, and each size comes with different specifications, including CPU, memory, and network bandwidth. When you create a VM, you choose a specific size based on your application's requirements. It's crucial to select a VM size that offers sufficient network bandwidth for your workload. You can read about the network bandwidth of the different VM sizes here https://learn.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series

VM Disk Performance:

Slow upload speeds could be related to disk I/O. If you are uploading or downloading data to/from disks on the VM, ensure that the disk configuration (example: Standard HDD, Standard SSD, Premium SSD) meets your performance requirements. Check this link for more information about disk sizes and performance https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types

Network Monitoring:

Use Azure Monitor to monitor network performance and identify any bottlenecks or issues. Look for network metrics in the Azure Portal or use Azure Monitor logs.

Network Security Group Rules:

Review your Network Security Group (NSG) rules. Ensure that you are not restricting the necessary traffic and that the rules are appropriately configured.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-14T09:28:04.84+00:00

@Dalbir Singh /IT /DB

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

May I ask how the Azure VM and OnPrem users are connected?

Is it via Internet ?

Are you uploading
from Azure VM to OnPrem via Internet?
or from Azure VM to an Internet Site?
or from OnPrem server to Azure VM via Internet?

May I ask where the firewall comes in?

Is it placed in Azure or in OnPrem

Can you please share the network flow on the upload.

Please note that in cases where traffic is uploaded via Internet, no bandwidth can be guaranteed as Azure has little control over the underlying Internet infrastructure.

Thanks,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-15T05:24:58.6666667+00:00

Dear All

Thanks to everyone for the valuable suggestions.

Our setup is in North India, and the Azure region selected is in Central India.

Azure VM and OnPrem users connected with the internet.

Upload and download flow is via the internet. The connection was established with the WAN IP configured in the firewall.

Firewall is configured on Prem; only VMs are configured in Azure Portal.

Earlier, it was working fine, but since the last two months, we have been getting drops while connecting to On Prem systems with Azure VM.

Do we require any changes in our Azure tunnel or firewall! Something is missing from our end?
Dana Akary 0 Reputation points

2023-11-16T07:59:29.94+00:00

If you connect to the VM from outside the company network do you still experience the same?
Dalbir Singh /IT /DB 0 Reputation points

2023-11-16T08:49:39.8166667+00:00

While connecting with the VPN client from different internet connection, file copy takes time; the 340 MB file size shows 1 hour, 30 minutes of time remaining.
Dana Akary 0 Reputation points

2023-11-16T09:18:14.9933333+00:00

Another test you can try to further isolate the issue, If you have a different VM in the same network try to upload a file with the same size to the affected VM.

Monitor as well the CPU and Disk performance of the VM itself during the copy process.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-16T09:37:09.2+00:00

@Dalbir Singh /IT /DB

Thanks for sharing your environment.

Since the connection takes place via Internet, we cannot comment much on the latency or bandwidth (thus impacting the performance).

As stated by Dana Akary, you can try to reproduce this issue from a similar VM in the same VNET as this VM and isolate if this is specific to a VM or to the entire VNET.

You can try using SFTP or FTP Clients such as WinScp and check if the performance improves.

Cheers,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-17T05:02:12.2966667+00:00

Thanks everyone for your valuable time & Suggestion.

One strange thing we noticed in this activity. while pinging our Azure VM Server Public IP, No such drops reported. Alongside while pinging our local private IP we are getting drops & latency. Somewhere around in 3 digit.

Reply from 10.*** * *: bytes=32 time=263ms TTL=128

Reply from 10.*** * *: bytes=32 time=151ms TTL=128

Reply from 10.*** * *: bytes=32 time=414ms TTL=128

Reply from 10.*** * *: bytes=32 time=144ms TTL=128

Do we need any changes, Please let us know .........

Thanks again.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-20T11:42:28.69+00:00

@Dalbir Singh /IT /DB

There is an ongoing issue on latency of Azure VPN gateway : https://learn.microsoft.com/en-us/answers/questions/1383683/azure-vpn-latency-every-8-12-seconds.

We have created a document item to call out the latency issue :

https://github.com/MicrosoftDocs/azure-docs/issues/116873

To troubleshoot latency issues, We will need a specialized 1:1 session, where a support engineer can have a screen share session to pinpoint the issue. If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Dalbir Singh /IT /DB 0 Reputation points

2023-11-21T06:33:00.8366667+00:00

@KapilAnanth-MSFT

Thanks for the update.

We are working on the issue with the Azure support engineer. Hopefully, we will resolve the issue timely.
KapilAnanth-MSFT 27,831 Reputation points Microsoft Employee

2023-11-21T11:40:37.38+00:00

@Dalbir Singh /IT /DB

Thanks for the update.

Hopefully, the issue gets resolved soon.

Cheers,

Kapil.
Dana Akary 0 Reputation points

2023-11-22T11:36:58.8133333+00:00

Please try the steps explained here https://learn.microsoft.com/en-us/azure/vpn-gateway/reset-gateway to reset the VPN Gateway? Lets see the ping results and upload speed after the reset.

Azure VM Server Comunication problem with On premises computers

Activity