unable to access internet using the additional interface in azure VM.

Vigneshwaran S 31

Hi,

I attached an additional interface(eth1) with attached public IP in the VM. I need to access the internet through the additional interface. I tried creating the default gateway for that interface ( sudo ip route add default via 10.x.x.x dev eth1 ). After run that command, the VM disconnects the ssh. I need to restart that VM and it goes back to default values.

Is there any other configuration needed for this?

VM: ubuntu 18.04 server
Both subnets and interfaces in the same VM
NSGs are configured to allow external traffic

route -n
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.1.0.1 0.0.0.0 UG 100 0 0 eth0
10.1.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eth0
10.2.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

Vigneshwaran S 31

sudo lshw -class network (I have many network as I have k8s pods, Here are some useful network)

azureuser@open5gs-calico:~$ sudo lshw -class network
  *-network
       description: Ethernet interface
       product: MT27710 Family [ConnectX-4 Lx Virtual Function]
       vendor: Mellanox Technologies
       physical id: 2
       bus info: pci@b4ab:00:02.0
       logical name: enP46251s1
       version: 80
       serial: 60:45:bd:73:87:68
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress msix bus_master cap_list ethernet physical autonegotiation
       configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=5.0-0 firmware=14.30.1210 (MSF0010110035) latency=0 link=yes multicast=yes slave=yes
       resources: iomemory:f0-ef irq:0 memory:fe0000000-fe00fffff
  *-network:0
       description: Ethernet interface
       physical id: 1
       logical name: eth0
       serial: 60:45:bd:73:87:68
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=hv_netvsc duplex=full firmware=N/A ip=10.1.0.4 link=yes multicast=yes
 *-network:9
       description: Ethernet interface
       physical id: a
       logical name: eth1
       serial: 60:45:bd:ac:68:0f
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=hv_netvsc duplex=full firmware=N/A ip=10.2.0.4 link=yes multicast=yes

Vigneshwaran S 31

sudo lshw -class network -short

azureuser@open5gs-calico:~$ sudo lshw -class network -short
H/W path      Device           Class      Description
=====================================================
/0/2          enP46251s1       network    MT27710 Family [ConnectX-4 Lx Virtual Function]
/1            eth0             network    Ethernet interface
/2            cali869e458a7ef  network    Ethernet interface
/3            caliafaadae72aa  network    Ethernet interface
/4            cali4f3fd49ef57  network    Ethernet interface
/5            cali38c7412e17e  network    Ethernet interface
/6            cali720e3391092  network    Ethernet interface
/7            cali1ed0f76a640  network    Ethernet interface
/8            calief154796485  network    Ethernet interface
/9            cali6e50e7dbe1c  network    Ethernet interface
/a            eth1             network    Ethernet interface
/b            cali7aef93154e3  network    Ethernet interface
/c            calid7b09fa3157  network    Ethernet interface
/d            cali7240ef20261  network    Ethernet interface
/e            cali054444db945  network    Ethernet interface
/f            cali4caa781661d  network    Ethernet interface
/10           calif0c669c0ade  network    Ethernet interface

Vigneshwaran S 31

sudo lshw -class network -short

azureuser@open5gs-calico:~$ sudo lshw -class network -short
H/W path      Device           Class      Description
=====================================================
/0/2          enP46251s1       network    MT27710 Family [ConnectX-4 Lx Virtual Function]
/1            eth0             network    Ethernet interface
/2            cali869e458a7ef  network    Ethernet interface
/3            caliafaadae72aa  network    Ethernet interface
/4            cali4f3fd49ef57  network    Ethernet interface
/5            cali38c7412e17e  network    Ethernet interface
/6            cali720e3391092  network    Ethernet interface
/7            cali1ed0f76a640  network    Ethernet interface
/8            calief154796485  network    Ethernet interface
/9            cali6e50e7dbe1c  network    Ethernet interface
/a            eth1             network    Ethernet interface
/b            cali7aef93154e3  network    Ethernet interface
/c            calid7b09fa3157  network    Ethernet interface
/d            cali7240ef20261  network    Ethernet interface
/e            cali054444db945  network    Ethernet interface
/f            cali4caa781661d  network    Ethernet interface
/10           calif0c669c0ade  network    Ethernet interface

Vigneshwaran S 31 Reputation points

2022-03-11T06:49:24.03+00:00

azureuser@open5gs-calico:~$ ip a show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 60:45:bd:73:87:68 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.4/22 brd 10.1.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::6245:bdff:fe73:8768/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 60:45:bd:ac:68:0f brd ff:ff:ff:ff:ff:ff
inet 10.2.0.4/24 brd 10.2.0.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::6245:bdff:feac:680f/64 scope link
valid_lft forever preferred_lft forever
4: enP46251s1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master eth0 state UP group default qlen 1000
link/ether 60:45:bd:73:87:68 brd ff:ff:ff:ff:ff:ff

Vigneshwaran S 31

ip a s eth0

azureuser@open5gs-calico:~$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:45:bd:73:87:68 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.4/22 brd 10.1.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6245:bdff:fe73:8768/64 scope link
       valid_lft forever preferred_lft forever
azureuser@open5gs-calico:~$ ip a s eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:45:bd:ac:68:0f brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.4/24 brd 10.2.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::6245:bdff:feac:680f/64 scope link
       valid_lft forever preferred_lft forever

Vigneshwaran S 31

azureuser@open5gs-calico:~$ netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
docker0   1500        0      0      0 0             0      0      0      0 BMU
enP46251  1500    15333      0      0 0        299178      0      0      0 BMsRU
eth0      1500   218079      0      0 0        278692      0      0      0 BMRU
eth1      1500     7056      0      0 0           167      0      0      0 BMRU
lo       65536  8545129      0      0 0       8545129      0      0      0 LRU
tunl0     1480        0      0      0 0             0      0      0      0 ORU

5 answers

Alan Kinane 16,811 Reputation points MVP

2022-03-10T15:17:02.353+00:00

Here is the process for adding a secondary network interface for Ubuntu 18.04: https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/virtual-network-multiple-ip-addresses-portal#linux-ubuntu-1804
Please sign in to rate this answer.

1 person found this answer helpful.
Vigneshwaran S 31 Reputation points

2022-03-11T03:07:33.763+00:00

Hi, Any update on this issue?

risolis 8,711 Reputation points

2022-03-11T03:22:27.953+00:00

Hi @Vigneshwaran S

Do you have your NSG at the Vnet level or NIC level?

If you have your NSGs per NIC level, please check effective routes as well as effective security rules....

Vigneshwaran S 31 Reputation points

2022-03-11T04:27:14.63+00:00

Hi,

I attached NSG for every NIC. So, the scope is per NIC level.

This is my effective security rules

And ,this is first 5 rows of effective route table
Sign in to comment
risolis 8,711 Reputation points

2022-03-11T06:57:07.897+00:00

@Vigneshwaran S

You have been started in the beginning with the following details hehehe:

(I have many network as I have k8s pods, Here are some useful network)

Are you using CNI advanced networking features on your K8 node?

Thx!
Please sign in to rate this answer.

1 person found this answer helpful.
Vigneshwaran S 31 Reputation points

2022-03-11T07:01:06.733+00:00

I am using Calico CNI in k8s node.

I checked, After adding the default route for secondary interface (sudo ip route add default via 10.x.x.x dev eth1). I can able to ping that interface from other VMs. However I can't access the host VM, it gets automatically disconnected.

risolis 8,711 Reputation points

2022-03-11T07:03:17.83+00:00

VXLAN right?

Vigneshwaran S 31 Reputation points

2022-03-11T07:18:34.12+00:00

calico config file

{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "info",
"log_file_path": "/var/log/calico/cni/cni.log",
"datastore_type": "kubernetes",
"nodename": "open5gs-calico",
"mtu": 0,
"ipam": {
"type": "calico-ipam"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {"bandwidth": true}
}
]
}
"/

risolis 8,711 Reputation points

2022-03-11T07:32:02.7+00:00

Have you tried to do it on the following way?

configuring ssh local port-forwarding...

Cheers!

Vigneshwaran S 31 Reputation points

2022-03-11T07:46:38.79+00:00

No, I haven't done the ssh port forwarding

Vigneshwaran S 31 Reputation points

2022-03-12T02:33:03.787+00:00

Hello @risolis

Any update on this or any documentation on how to access internet through multiple interfaces in a Azure VM?
If so please let me know.

Thankyou.

risolis 8,711 Reputation points

2022-03-12T17:52:37.94+00:00

By default, in Linux, if multiple NICs are in same subnet, all traffics will route through the default NIC that usually will be eth0, if multiple NICs are added, to route traffics back to the NIC that receives it, we need to create route table for that NIC....

risolis 8,711 Reputation points

2022-03-12T18:04:46.73+00:00

@Vigneshwaran S

Just in case when you made what I mentioned to you and if you test it let me know.... how it goes because that should work like that and if not, check on the primary interface is IP forwarding is enabled as well.

Regards!
Sign in to comment
risolis 8,711 Reputation points

2022-03-11T04:56:31.903+00:00

@Vigneshwaran S

Your virtual network is 10/8 as per your output(Correct me if I am mistaken)....

Did you subnet your space address into small subnets or not? You can do subnetting and allocate each NIC in different subnet....

Did you restart your VM after adding another NIC?
Do you have any UDR(User define routes set up)?
Please sign in to rate this answer.
Vigneshwaran S 31 Reputation points

2022-03-11T05:04:36.66+00:00

@risolis

Yes, My virtual network is 10.0.0.0/8

Yes, I subnet my space address in to small subnets(ex: eth0 uses 10.1.0.0/24 and eth1 uses 10.2.0.0/24)

Yes, I restarted vm VM after added another NIC

No, I don't have any UDR

risolis 8,711 Reputation points

2022-03-11T05:53:01.77+00:00

May I have the following outputs please?

mesg | grep -i eth0 >>State the ethernet interface using the one described before..
netstat -i
sudo lshw -class network
sudo lshw -class network -short
ip a show
ip a s eth0 >>State the ethernet interface using the one described before..

Regards,

Vigneshwaran S 31 Reputation points

2022-03-11T06:38:21.363+00:00

okay sure

azureuser@open5gs-calico:~$ netstat -i Kernel Interface table Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg docker0 1500 0 0 0 0 0 0 0 0 BMU enP46251 1500 15333 0 0 0 299178 0 0 0 BMsRU eth0 1500 218079 0 0 0 278692 0 0 0 BMRU eth1 1500 7056 0 0 0 167 0 0 0 BMRU lo 65536 8545129 0 0 0 8545129 0 0 0 LRU tunl0 1480 0 0 0 0 0 0 0 0 ORU
Sign in to comment
risolis 8,711 Reputation points

2022-03-14T07:32:46.143+00:00

@Vigneshwaran S

I just wanted to know if you have any other concern.

Regards,
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Vigneshwaran S 31 Reputation points

2022-04-19T13:52:33.3+00:00

Hi,

Sorry for the late reply.

I followed this blog (https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System) and it is working.

Thankyou @risolis and @Alan Kinane .
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

unable to access internet using the additional interface in azure VM.

5 answers