connecting to the wireserver on secondary/VRF interfaces

Question

connecting to the wireserver on secondary/VRF interfaces

Peter Morrow 91

Hello,

On our VM (running a custom version of a debian based OS) I can communicate with the wire server over the primary interface which is in the default vrf:

gnos@vEdge:~$ ip route show  
default via 10.0.0.1 dev eth0   
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4   
168.63.129.16 via 10.0.0.1 dev eth0   
169.254.169.254 via 10.0.0.1 dev eth0

Talking to the wireserver:

gnos@vEdge:~$ curl --fail -v -X 'GET' -H "x-ms-agent-name: azure-vm-register" -H "Content-Type: text/xml;charset=utf-8" -H "x-ms-version: 2012-11-30" "http://168.63.129.16/machine/?comp=goalstate" Note: Unnecessary use of -X or --request, GET is already inferred.  
*   Trying 168.63.129.16:80...  
* Connected to 168.63.129.16 (168.63.129.16) port 80 (#0)  
> GET /machine/?comp=goalstate HTTP/1.1  
> Host: 168.63.129.16  
> User-Agent: curl/7.74.0  
> Accept: */*  
> x-ms-agent-name: azure-vm-register  
> Content-Type: text/xml;charset=utf-8  
> x-ms-version: 2012-11-30  
>   
* Mark bundle as not supporting multiuse  
< HTTP/1.1 200 OK  
< Content-Type: text/xml; charset=utf-8  
< Server: Microsoft-IIS/10.0  
< Date: Sat, 20 Aug 2022 05:54:55 GMT  
< Content-Length: 1865  
<   
<?xml version="1.0" encoding="utf-8"?>  
<GoalState xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="goalstate10.xsd">  
  <Version>2012-11-30</Version>  
  <Incarnation>1</Incarnation>  
  <Machine>  
    <ExpectedState>Started</ExpectedState>  
    <StopRolesDeadlineHint>300000</StopRolesDeadlineHint>  
    <LBProbePorts>  
      <Port>16001</Port>  
    </LBProbePorts>  
    <ExpectHealthReport>FALSE</ExpectHealthReport>  
  </Machine>  
  <Container>  
    <ContainerId>f7525b44-b0c2-4e9b-9a4a-e746824e6110</ContainerId>  
    <RoleInstanceList>  
      <RoleInstance>  
        <InstanceId>f10774f1-c502-4c60-ac63-89dcb6e84492._vEdge</InstanceId>  
        <State>Started</State>  
        <Configuration>  
          <HostingEnvironmentConfig>http://168.63.129.16:80/machine/f7525b44-b0c2-4e9b-9a4a-e746824e6110/f10774f1%2Dc502%2D4c60%2Dac63%2D89dcb6e84492.%5FvEdge?comp=config&amp;type=hostingEnvironmentConfig&amp;incarnation=1</HostingEnvironmentConfig>  
          <SharedConfig>http://168.63.129.16:80/machine/f7525b44-b0c2-4e9b-9a4a-e746824e6110/f10774f1%2Dc502%2D4c60%2Dac63%2D89dcb6e84492.%5FvEdge?comp=config&amp;type=sharedConfig&amp;incarnation=1</SharedConfig>  
          <ExtensionsConfig>http://168.63.129.16:80/machine/f7525b44-b0c2-4e9b-9a4a-e746824e6110/f10774f1%2Dc502%2D4c60%2Dac63%2D89dcb6e84492.%5FvEdge?comp=config&amp;type=extensionsConfig&amp;incarnation=1</ExtensionsConfig>  
          <FullConfig>http://168.63.129.16:80/machine/f7525b44-b0c2-4e9b-9a4a-e746824e6110/f10774f1%2Dc502%2D4c60%2Dac63%2D89dcb6e84492.%5FvEdge?comp=config&amp;type=fullConfig&amp;incarnation=1</FullConfig>  
          <ConfigName>f10774f1-c502-4c60-ac63-89dcb6e84492.0.f10774f1-c502-4c60-ac63-89dcb6e84492.0._vEdge.1.xml</ConfigName>  
        </Configuration>  
      </RoleInstance>  
    </RoleInstanceList>  
  </Container>  
* Connection #0 to host 168.63.129.16 left intact  
</GoalState>gnos@vEdge:~$

However I have a requirement that means I need to do this on another interface which is in a VRF, so I added the static routes to allow the VRF to communicate with the wireserver:

gnos@vEdge:~$ ip route show vrf vrf2  
default via 10.0.1.1 dev vpp17 proto kernel-mgr metric 512   
unreachable default metric 4278198272   
10.0.1.0/24 dev vpp17 proto kernel scope link src 10.0.1.4   
10.0.1.0/24 dev vpp17 proto kernel-mgr scope link metric 512   
127.0.0.0/8 dev vrf2 proto kernel scope link src 127.0.0.1   
168.63.129.16 via 10.0.1.1 dev vpp17   
169.254.169.254 via 10.0.1.1 dev vpp17   
gnos@vEdge:~$

However I cannot communicate with the wireserver:

gnos@vEdge:~$ curl --fail -v -X 'GET' -H "x-ms-agent-name: azure-vm-register" -H "Content-Type: text/xml;charset=utf-8" -H "x-ms-version: 2012-11-30" "http://168.63.129.16/machine/?comp=goalstate" --interface vrf2  
Note: Unnecessary use of -X or --request, GET is already inferred.  
*   Trying 168.63.129.16:80...  
^C  
gnos@vEdge:~$

This is not a routing issue as I can do other operations via that VRF:

gnos@vEdge:~$ ping -I vrf2 www.google.comping: Warning: source address might be selected on device other than: vrf2  
PING www.google.com (172.253.63.147) from 10.0.1.4 vrf2: 56(84) bytes of data.  
64 bytes from bi-in-f147.1e100.net (172.253.63.147): icmp_seq=1 ttl=53 time=5.91 ms  
64 bytes from bi-in-f147.1e100.net (172.253.63.147): icmp_seq=2 ttl=53 time=5.98 ms  
64 bytes from bi-in-f147.1e100.net (172.253.63.147): icmp_seq=3 ttl=53 time=5.90 ms  
64 bytes from bi-in-f147.1e100.net (172.253.63.147): icmp_seq=4 ttl=53 time=6.05 ms  
^C  
--- www.google.com ping statistics ---  
4 packets transmitted, 4 received, 0% packet loss, time 3005ms  
rtt min/avg/max/mdev = 5.896/5.959/6.050/0.061 ms  
gnos@vEdge:~$

When trying to communicate with the wireserver in a VRF I observed via packet capture that packets are being sent out, they are not being received back however.

gnos@vEdge:~$ sudo tcpdump -i vrf2  
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode  
listening on vrf2, link-type EN10MB (Ethernet), snapshot length 262144 bytes  
06:01:15.280748 IP 10.0.1.4.59698 > 168.63.129.16.http: Flags [S], seq 900444532, win 64240, options [mss 1460,sackOK,TS val 2926790123 ecr 0,nop,wscale 7], length 0  
06:01:16.293418 IP 10.0.1.4.59698 > 168.63.129.16.http: Flags [S], seq 900444532, win 64240, options [mss 1460,sackOK,TS val 2926791136 ecr 0,nop,wscale 7], length 0  
06:01:18.309415 IP 10.0.1.4.59698 > 168.63.129.16.http: Flags [S], seq 900444532, win 64240, options [mss 1460,sackOK,TS val 2926793152 ecr 0,nop,wscale 7], length 0  
06:01:22.437473 IP 10.0.1.4.59698 > 168.63.129.16.http: Flags [S], seq 900444532, win 64240, options [mss 1460,sackOK,TS val 2926797280 ecr 0,nop,wscale 7], length 0  
06:01:30.629472 IP 10.0.1.4.59698 > 168.63.129.16.http: Flags [S], seq 900444532, win 64240, options [mss 1460,sackOK,TS val 2926805472 ecr 0,nop,wscale 7], length 0

Are there any restrictions on which interfaces can communicate with the wireserver? The interface which is failing here in not the primary and has AN enabled. Maybe the azure host is filtering on MAC address of the primary interface?

Thanks

Peter

Accepted answer

1 additional answer

Your answer

Answer 1

KapilAnanth-MSFT 49,616 Microsoft Employee Moderator

Hi @Peter Morrow ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.
Apologies for the delay in my answer.

I understand that you are trying to connect with the Wireserver IP using a secondary IP address.
I am afraid that this is not a possible scenario.

WireServer IP and Metadata Service IPs are platform dependent and can be reached only via primary IP.

https://learn.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service?tabs=linux#frequently-asked-questions

Please let me know if you have further queries on this.

Cheers,
Kapil

----------------------------------------------------------------------------------------------------------------

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Peter Morrow 91 Reputation points

2022-08-24T08:27:39.207+00:00

Thanks Kapil and sorry for missing this in the docs!

Peter.
KapilAnanth-MSFT 49,616 Reputation points Microsoft Employee Moderator

2022-08-24T09:51:21.687+00:00

Hi @Peter Morrow

You are most welcome.

Cheers,
Kapil

Answer 2

risolis 8,741

Hello @Peter Morrow

Thank you for your post.

I wonder if you had enabled for this case scenario this feature "IP forwarding"...

Furthermore, I would like to know if when you tried to ping any destination target.... Were you doing this ping with the source syntax or not?

Finally, If you did a traceroute what the final result/output was for this?

Looking forward to your feedback,

Cheers,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

risolis 8,741 Reputation points

2022-08-20T22:09:34.073+00:00

Hi @Peter Morrow

Also, I just wanted to ask if you multiple NIC's on this environment or just a single NIC?

Looking forward to your feedback,

Cheers!
Peter Morrow 91 Reputation points

2022-08-21T07:24:53.917+00:00
Hi Ricardo,

Thanks for your help! Yes, there are multiple NICs. The issue I have is communicating with the wireserver on the 2nd interface which is also in a linux VRF.

IP Forwarding is indeed enabled on this interface as well as accelerated networking. Pings appear to be blocked on the path to the wireserver, so I cannot ping it even in the default VRF, same for traceroute which uses ICMP echo's.

All pings and curl commands are using the source syntax of the source interface to use, for example (vpp17) is the vrf interface:

gnos@vEdge:~$ ping -I vpp17 8.8.8.8PING 8.8.8.8 (8.8.8.8) from 10.0.1.4 vpp17: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=53 time=5.16 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=53 time=5.22 ms ^C --- 8.8.8.8 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 5.156/5.186/5.216/0.030 ms gnos@vEdge:~$

So from the above you can see that connectivity is fine on both interfaces, but for some reason I get no replies from the wireserver when using the 2nd interface (called vpp17). This leads me to believe that the azure host has some additional rules on what can connect to the wireserver.

Thanks,
Peter.
risolis 8,741 Reputation points

2022-08-22T01:28:15.883+00:00

Hello @Peter Morrow

Many thanks for extra info which wonderful to get... : )

Now I wonder if I can get the following outputs if possible:

ip -d link show type vrf
ip -br link show type vrf
ip link show vrf
neigh show vrf

Furthermore, I have a couple of question for you which are the ones below:

Were any iif and oif rules configured?

Did you bind any socket to any particular VRF?

Looking forward to your feedback,

Cheers,
Peter Morrow 91 Reputation points

2022-08-22T08:18:15.293+00:00

Hi Ricardo,

I think I can simplify this problem a bit by taking the whole VRF stuff out of it, instead consider this example and repro:

We have a debian 11 VM, with 2 interfaces:

address space: 10.1.0.0/16
interface 1 subnet 10.1.0.0/24.
interface 1 address: 10.1.0.4

interface 2 subnet: 10.1.1.0/24
interface 2 address: 10.1.1.4.
inerface 2: AN enabled, ip forwarding enabled.

If we boot the VM then DHCP runs for both inerfaces, the routing table looks like this:

azureuser@deb3:~$ ip route show
default via 10.1.0.1 dev eth0
10. 1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.4
10. 1.1.0/24 dev eth1 proto kernel scope link src 10.1.1.4
168. 63.129.16 via 10.1.0.1 dev eth0
169. 254.169.254 via 10.1.0.1 dev eth0
azureuser@deb3:~$

Connecting to the wireserver on eth0 works:

azureuser@deb3:~$ curl --fail -v -X 'GET' -H "x-ms-agent-name: azure-vm-register" -H "Content-Type: text/xml;charset=utf-8" -H "x-ms-version: 2012-11-30" "http://168.63.129.16/machine/?comp=goalstate" --interface eth0 &>/dev/null
azureuser@deb3:~$ echo $?
0
azureuser@deb3:~$

Now to connect via eth1 I need to change the routing table a bit since the default route above is via eth0:

Will continue since I am running out of characters...
Peter Morrow 91 Reputation points

2022-08-22T08:58:06.997+00:00

Continued...

azureuser@deb3:~$ sudo ip route del 168.63.129.16
azureuser@deb3:~$ sudo ip route add 168.63.129.16 via 10.1.1.1 dev eth1
azureuser@deb3:~$ ip route show
default via 10.1.0.1 dev eth0
10. 1.0.0/24 dev eth0 proto kernel scope link src 10.1.0.4
10. 1.1.0/24 dev eth1 proto kernel scope link src 10.1.1.4
168. 63.129.16 via 10.1.1.1 dev eth1
169. 254.169.254 via 10.1.0.1 dev eth0
azureuser@deb3:~$

So now we should be able to reach 168.63.129.16 via eth1 but we cannot:

azureuser@deb3:~$ curl --fail -v -X 'GET' -H "x-ms-agent-name: azure-vm-register" -H "Content-Type: text/xml;charset=utf-8" -H "x-ms-version: 2012-11-30" "http://168.63.129.16/machine/?comp=goalstate" --interface eth1 &>/dev/null
^C
azureuser@deb3:~$ echo $?
130
azureuser@deb3:~$

It fails!

I don't have any rules in the NSG which would block outbound or inbound traffic.

azureuser@deb3:~$ ip neigh
10. 1.0.1 dev eth0 lladdr 12:34:56:78:9a:bc REACHABLE
10. 1.1.1 dev eth1 lladdr 12:34:56:78:9a:bc REACHABLE
azureuser@deb3:~$

Curl/ping is the one doing the socket binding.

azureuser@deb3:~$ ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:df:ab:99 brd ff:ff:ff:ff:ff:ff
azureuser@deb3:~$ ip link show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:0d:3a:e6:0f:ed brd ff:ff:ff:ff:ff:ff
azureuser@deb3:~$
risolis 8,741 Reputation points

2022-08-22T16:45:30.993+00:00

Let me check it and get back to you shortly

Thanks
Peter Morrow 91 Reputation points

2022-08-22T16:47:59.913+00:00

Thanks Ricardo,

Some more info from todays testing. I was able to talk to the wireserver from a vrf today but it seems source interface must be the primary azure interface, the common factor is the failing cases is I cannot talk to the wireserver from non primary interfaces.

So I am good for now but it would be interesting to confirm that VMs can only communicate to the wireserver on the primary interface.

Cheers,
Peter.
risolis 8,741 Reputation points

2022-08-23T09:39:16.57+00:00

Sure.

Let me double-check it and I shall give you my findings.

Many thanks!
risolis 8,741 Reputation points

2022-08-24T05:59:10.92+00:00

Hi @Peter Morrow

Many thanks for the chance to participate on this case scenario.

Here are my 2 cents regarding to this unexpected behavior observed from you when setting this up.

Please read the following article below:

https://www.tenable.com/audits/items/CIS_Debian_Linux_8_Server_v2.0.2_L1.audit:bb0f399418f537997c2b44741f2cd634

I hope that was helpful for you to have a better understanding about it.

Have a good one : )

Cheers,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Share via

connecting to the wireserver on secondary/VRF interfaces

1 additional answer

Your answer