Private IP addresses in public ICMP packets of virtual machines

Question

Private IP addresses in public ICMP packets of virtual machines

Johannes Morgenroth 0

I think I have found a bug in the default NAT of virtual networks in combination with public IP addresses. In my setup, I see that ICMP messages due to errors, e.g. port unreachable or need to frag, are not referenced right with virtual machines exposed via public IP address. Instead of the public IP address, the internal private IP address is referenced in ICMP packets. Can somebody confirm that this is a bug, intended behavior, or unsupported for some reason? In my case the issue "just" leads to performance issues. But I think that the exposure of internal structures of the Azure setups might be a security issue.

Long story

ICMP responses on errors usually carry three IP addresses. The source, destination and the destination of the origin IP packet. Since in Azure public IP addresses are translated to private IP addresses and back (NAT), this is also necessary for ICMP packets to and from the virtual machine. I observed that source and destination are translated correctly, but the destination of the origin IP packet is not changed and sticks to the private IP address.

Example with unreachable port

In this example, I replaced the IP ranges for privacy reasons. So please don't be confused. ;-) The setup is as follows: A virtual machine connected to a virtual network with a private subnet (1.1.1.0/24). A public IP address (2.2.2.2) is associated to the virtual network interface (1.1.1.4) of the virtual machine. The Internet router on the client side has the public IP address 9.9.9.9. To make the issue visible, we send a UDP packet from the client side to an unreachable port, here 2.2.2.2:6000. The network security group is configured to accept packets on that port and passes the packet to the host. On the host, we see the packet in tcpdump and the host replies with an ICMP unreachable packet. The destination of the origin IP packet and the source are equal in this case.

13:56:38.901679 IP 9.9.9.9.54123 > 1.1.1.4.6000: UDP, length 1450
13:56:38.901735 IP 1.1.1.4 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556
13:56:39.903037 IP 9.9.9.9.54123 > 1.1.1.4.6000: UDP, length 1450
13:56:39.903088 IP 1.1.1.4 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556
13:56:40.904540 IP 9.9.9.9.54123 > 1.1.1.4.6000: UDP, length 1450
13:56:40.904598 IP 1.1.1.4 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556

On the inbound interface of the client side router, we see that the source has changed as expected but the destination of the origin IP packet didn't. It refers to the internal private IP address within the Azure private subnet. Instead of 1.1.1.4 it should be 2.2.2.2.

14:56:38.911702 IP 2.2.2.2 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556
14:56:39.915429 IP 2.2.2.2 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556
14:56:40.914425 IP 2.2.2.2 > 9.9.9.9: ICMP 1.1.1.4 udp port 6000 unreachable, length 556

Other experiments

The same happens with ICMP messages regarding fragmentation. The address in the IP header is masqueraded but the reference in the ICMP packet stays the private IP address of the virtual network.

13:49:26.695277 IP 1.1.1.68 > 9.9.9.9: ICMP 1.1.1.68 unreachable - need to frag (mtu 1437), length 556
13:49:27.696577 IP 1.1.1.68 > 9.9.9.9: ICMP 1.1.1.68 unreachable - need to frag (mtu 1437), length 556

ChaitanyaNaykodi-MSFT 27,481 Reputation points Microsoft Employee Moderator

2024-01-19T04:31:31.5033333+00:00

@Johannes Morgenroth
Thank you for reaching out. Based on your question above In ICMP protocol the Destination unreachable is generated by the host. Can you please let me know the security concern here if the internal private IP is exposed, so that I can provide an answer accordingly? Thank you!
Johannes Morgenroth 0 Reputation points

2024-01-19T09:52:07.2133333+00:00

At the moment I am not really worried about the implications of exposing. But I think that knowing internal structure can support an attack as second vector. In general, you're right that just knowing the internal address isn't enough to compromise a host. However, I am more concerned about that the ICMP errors do not reach the clients as shown in section "other experiements". I have chosen the unreachable port error as first explanation because of it is easier to replicate.

1 answer

Your answer

ChaitanyaNaykodi-MSFT 27,481 Reputation points Microsoft Employee Moderator

2024-01-19T04:31:31.5033333+00:00

@Johannes Morgenroth
Thank you for reaching out. Based on your question above In ICMP protocol the Destination unreachable is generated by the host. Can you please let me know the security concern here if the internal private IP is exposed, so that I can provide an answer accordingly? Thank you!
Johannes Morgenroth 0 Reputation points

2024-01-19T09:52:07.2133333+00:00

At the moment I am not really worried about the implications of exposing. But I think that knowing internal structure can support an attack as second vector. In general, you're right that just knowing the internal address isn't enough to compromise a host. However, I am more concerned about that the ICMP errors do not reach the clients as shown in section "other experiements". I have chosen the unreachable port error as first explanation because of it is easier to replicate.

Answer 1

Silvia Wibowo 6,046 Microsoft Employee Volunteer Moderator

Hi @Johannes Morgenroth , I understand that you are concerned about private IPv4 address in the content of ICMP error message from an Azure VM.

Let's clarify a few things:

Private IPv4 addresses (RFC 1918) are not publicly routable on the internet. Knowing that a server's private IPv4 address is 10.0.0.4 does not increase its attack point from public network (internet).
You set up your Azure environment to allow traffic (NSG open port 6000) to your Azure VM, although your Azure VM does not listen to that port. Best practice would be to set up NSG to only allow inbound ports that applications in your Azure VM require.
ICMP error messages contain a data section that includes a copy of the entire IPv4 header, and it has a checksum to make sure that data is not altered. What I understood from your concern is that NAT process should also modify the content of ICMP error message, which is protected by checksum.

Please let me know if you have other concerns.

Johannes Morgenroth 0 Reputation points

2024-01-19T08:57:40.0266667+00:00
Hi @Silvia Wibowo , thanks for your reply.

At the moment I am not really worried about the implications of exposing. But I think that knowing internal structure can support an attack as second vector. In general, you're right that just knowing the internal address isn't enough to compromize a host.

Yes, thats true for a clean setup. But it might comes the case that a service is broken or crashed by an attacker. However, I am more concerned about that the ICMP errors do not reach the clients as shown in section "other experiements". I have chosen the unreachable port error as first explanation because of it is easier to replicate.

Right, the IP addresses in the ICMP payload needs to be modified aswell. At least this is the case in state-of-the-art NAT mechanisms I know from Linux. Without the ICMP error will just be dropped because the client cannot correlate it with ongoing communication.

See https://www.rfc-editor.org/rfc/rfc5508, section 4.2.2. ICMP Error Packet Received from the Private Realm.

[...] the NAT device MUST use the matching NAT Session to translate the embedded packet.
Johannes Morgenroth 0 Reputation points

2024-01-19T09:50:10.88+00:00
Hi @Silvia Wibowo , thanks for your reply.

At the moment I am not really worried about the implications of exposing. But I think that knowing internal structure can support an attack as second vector. In general, you're right that just knowing the internal address isn't enough to compromize a host.

Yes, thats true for a clean setup. But it might comes the case that a service is broken or crashed by an attacker. However, I am more concerned about that the ICMP errors do not reach the clients as shown in section "other experiements". I have chosen the unreachable port error as first explanation because of it is easier to replicate.

Right, the IP addresses in the ICMP payload needs to be modified aswell. At least this is the case in state-of-the-art NAT mechanisms I know from Linux. Without the ICMP error will just be dropped because the client cannot correlate it with ongoing communication.

See https://www.rfc-editor.org/rfc/rfc5508, section 4.2.2. ICMP Error Packet Received from the Private Realm. [...] the NAT device MUST use the matching NAT Session to translate the embedded packet.
Silvia Wibowo 6,046 Reputation points Microsoft Employee Volunteer Moderator

2024-01-23T20:31:02.95+00:00

Hi @Johannes Morgenroth, network in cloud works very differently from on-premises network. Let's step back to your business/app requirement, rather than talking about protocol and NAT. What exactly do you need to achieve?
Johannes Morgenroth 0 Reputation points

2024-01-26T08:39:00.04+00:00

Hi @Silvia Wibowo , my business application has limitations on MTU it can process. For that reason, the deployment within the virtual network responds with "ICMP unreachable - need to frag". Since the embedded packet is not translated correctly, the client application does not receive the hint to reduce the MTU and packet loss is the consequence. This makes it impossible to build the use-case on top of Azure.
Silvia Wibowo 6,046 Reputation points Microsoft Employee Volunteer Moderator

2024-01-28T22:07:48.9833333+00:00

Hi @Johannes Morgenroth, rather than relying on ICMP error message, can you set the MTU on the client app or OS?
ChaitanyaNaykodi-MSFT 27,481 Reputation points Microsoft Employee Moderator

2024-01-29T20:57:28.5933333+00:00

@Johannes Morgenroth

Thank you for getting back, just following up here to see if you were able to see @Silvia Wibowo 's comment above. Thank you Silvia for the engagement here

In order to provide additional details on why MSS clamping is preferred.

As documented here. The PMTUD process is inefficient and affects network performance. When packets are sent that exceed a network path's MTU, the packets need to be retransmitted with a lower MSS. If the sender doesn't receive the ICMP Fragmentation Needed message, maybe because of a network firewall in the path (commonly referred to as a PMTUD blackhole), the sender doesn't know it needs to lower the MSS and will continuously retransmit the packet.

Hope this helps!

---------------Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Johannes Morgenroth 0 Reputation points

2024-01-31T17:31:55.77+00:00

Hi @Silvia Wibowo , that's impossible because the client OS nor the client app isn't fully controlled by us.
Johannes Morgenroth 0 Reputation points

2024-01-31T17:38:58.0866667+00:00

I am talking about UDP traffic. TCP tuning will not help. Instead of suggesting crude workarounds, I expected investigations on real solutions according to the standards from your side, see RFC 5508. However, I understand that MS Azure is not willing/capable in hosting our use-case for Bosch and we need to investigate other hosting services out there. I will report that to my manager. Thank you for your support so far.
ChaitanyaNaykodi-MSFT 27,481 Reputation points Microsoft Employee Moderator

2024-02-02T18:42:54.4133333+00:00

@Johannes Morgenroth
Thank you for getting back.It will help if you could file your feedback regarding this request on our feedback portal so that it can be tracked by the product team. Thank you! Chaitanya

Share via

Private IP addresses in public ICMP packets of virtual machines

Long story

Example with unreachable port

Other experiments

1 answer

Your answer