This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Today I was trying to do something that I thought would be very simple and it was to make my own NVA so devices from different VNets/subnets could reach the internet while maintaining traffic control through the NVA.
Currently I've set up 3 different VNets in a hub-and-spoke fashion. I've already configured VNet peering from the spokes to the hub and I was able to successfully communicate between spokes through the NVA using it as a default route for each spoke. Here are my routes:
At first, I set up the destination IP address/CIDR in a more specific way, defining the destination subnet at the other side instead of 0.0.0.0/0 and it worked nicely while maintaining internet connection. After that I configured it as you can see in the screenshots to use the NVA as my gateway to the internet but that’s when I started getting very confused.
With this configuration the NVA is not even able to see traffic that the spokes send over to the internet, only traffic between spokes. Here are some basic connectivity tests that I’ve made:
Traffic between spokes working perfectly:
Traffic sent to the internet is not even seen by the NVA:
I’ve read through the documentation that when you need to communicate to the internet through the NVA you must proxy or NAT internal traffic to your own NVA internal IP address, then it is sent to Azure’s default gateway but as you can see I’m not able to get any traffic from the spoke.
Just in case (because I can’t see any traffic) I set up some IP tables to NAT the source internal network to the NVA’s private address but it didn’t work either:
iptables -t nat -A POSTROUTING -s 10.1.0.0/24 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.2.0.0/24 -o eth0 -j MASQUERADE
I’ve also seen these requirements in the documentation, and I think I’ve them all, but I could be wrong:
Can someone please point me in the right direction? I’ve been trying so hard to make this work and after hours of trying different things I’m still not able to see any traffic from the spokes to the internet or make any successful connection to the internet at all.
Hi @Ariel Gonzalez
Did you allow traffic forwarder?
Can you review this article to compare it with your environment?
https://thewindowsupdate.com/2020/12/11/using-azure-firewall-as-a-network-virtual-appliance-nva/
Hello @Jackson Martins ,
Thank you for your answer. Yes, I've allowed traffic as described in the screenshot that you provided:
The only difference between the solution provided in that link is that I'm using a normal ubuntu machine as my NVA, which leads me to believe that my NAT configuration could be wrong, I'm basically doing Port Address Translation which is SNAT type instead of DNAT. I'm still confused because I think I should be able to at least see traffic coming from the spokes VNets with tcpdump.
Just to clarify, do you have 3 vnets right?
can you describe it to me?? like:
VNET 1
VNET 2
VNET 3 - NVA
VNET 1 <peering> VNET 3 <peering> VNET2
or
VNET 1 <peering> VNET 2
VNET 1 <peering> VNET 3
Yes, there are 3 VNets:
Branch-VNet-JapanEast -> Spoke, peering with Core-VNet-WestUS3
Branch-VNet-KoreaCentral -> Spoke, peering with Core-VNet-WestUS3
Core-VNet-WestUS3 -> Hub, NVA lives here
I was reviewing your screenshots and the ping to NVA is going but the traffic is not.
1 - did you see if there is some NSG blocking blocking the input?
2 - have you enabled network forwarding on the linux interface?
Just like:
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
Try to see the status for your custom routes going to Effective routes on the nic of VM:
see what the state of the route you created
Yes, IP forwarding is allowed in the NVA as well as in the azure portal:
But I'm not sure which NSG's should I be looking at, currently there are 3 NSG's, one in each VM. They are all using the default NSG rules when the VM is created:
1) VM in Branch-VNet-JapanEast:
2) VM in Branch-VNet-KoreaCentral:
3) NVA in Core-VNet-WestUS3:
On Core-VNet-WestUS3
Create a rule to allow all ports inbound from vnets (just to make sure)
And, what about effective routes in NIC from
Branch-VNet-JapanEast
And Branch-VNet-KoreaCentral
Go to Vm on vnet Branch-VNet-KoreaCentral
Open interface and click on effective routes,
Try to find your custom route for NVA, see the state.
Other test
Try to ping 1.1.1.1 not dns and paste the result
Alright I added the inbound rule to the NSG in the NVA Core-VNet-WestUS3 but rule 65000 should be allowing this traffic already:
Here are the effective routes for each VM:
1) Branch-JapanEast-TestVM
2) Branch-KoreaCentral-TestVM
3) Core-WestUS3-TestNVA
Here are the results after pinging 1.1.1.1, unfortunately there is still no traffic to be seen, not even an attempt to go to over the internet:
it looks like everything is ok
What the version of Linux are you using on NVA?
I have a similar scenario with Pfsensec I will try to replicate your scenario and paste de result here
Here is my distro information, it's an ubuntu machine:
Hi @Ariel Gonzalez
I have set up a similar environment and it is working normally, I will post some screenshots
I have 2 subnets
VNET-Jackson - 10.0.0.0/16 - 1 VM Windows -> 10.0.0.4
VNET-Linux - 10.2.0.0/16 - 1 VM Linux -> 10.2.0.4
My VNET peering
My route tables (to only VM windows on VNET Jackson)
Nexthop to VM LINUX
NSG for VM Linux:
IP Forwarding on NIC
]7
Ifconfig VM NIC linux
VM Iptables configuration
TCPDUMP src host VM windows:
PING VM Windows
Hi @Jackson Martins ,
Thank you for taking the time and replicating the scenario. After looking at your configuration I noticed that you allowed every traffic in the NVA's NSG. I went ahead and changed it to allow inbound ICMP from everywhere to everywhere and it worked! But I'm still confused why didn't I see any traffic with tcpdump coming from my internal machines? Do you have any theory? I think I should have seen at least packets coming in from my internal VM's but not going out, it would have been helpful to see that traffic for troubleshooting.
Hello @Ariel Gonzalez
Thank you for the post.
@Jackson Martins hi buddy : )
I just wanted to add a couple of observations for this case scenario.... One thing that brought my attention was that you were mentioning that you are you using an NVA(Network Virtual Appliance) so, based on your response this is your NVA "ubuntu machine as my NVA"
If this Ubuntu is acting as your NVA device and it is hosted as your HUB on this Hub-and-Spoke topology (Please any of your correct me if I am mistaken)
I wonder if this is set up as a Routing device or a Stateful/stateless Firewall... Which is one the role planned?
If it is acting as your FW device... I have 2 questions for this scenario which are:
Do you have security zones configured?
Do you configure any intra/inter security rule to allow traffic from different security zone? Or Any PBF rule??
Moreover, If it is your FW device or Router Device... Did you configure any VRF/Routing instance?
Finally, This other concern that I have is.... Have you checked your forwarding table as well?
Looking forward to your feedback,
Cheers,
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Hello @risolis ,
Thank you for your answer. For a second I thought that UFW could have been causing so much trouble but after checking there is nothing blocking any connection (as far as I know, there could be some default configuration that I'm not aware of), this NVA is currently supposed to be set up as routing only:
It doesn't have any VRF/Routing instance either, currently this NVA doesn't have any special configuration other than having IP Forwarding enabled.
EDIT: I forgot to mention that I had some IPTables doing PAT but didn't seem to work either, the NVA is not able to see any traffic sent over to the internet as I pointed out before.
Many thanks for your quick response.
I have a concern that I hope you can address..... I was taking a look at the IP allocation and I found that you are using this IP range 10.2.0.4/24.
Is that one the correct IP range that you want to use?
Looking forward to your feedback,
Cheers,
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Currently there are 3 VNets with the following address space:
1) Branch-VNet-JapanEast -> 10.1.0.0/16
2) Branch-VNet-KoreaCentral -> 10.2.0.0/16
3) Core-VNet-WestUS3 -> 10.0.0.0/16
Each VNet has 3 subnets, they all follow the same pattern of X.X.0.X/24, X.X.1.X/24, X.X.2.X/24. The IP that you're talking about is from Branch-KoreaCentral-TestVM / Subnet1. Spokes can reach each other but when I try to access anything on the internet not even the NVA is able to see this traffic, makes me wonder where those packets end up.
Many thanks.
Now I read one more time the whole thread....
This is the topology that you have as per your comments:
1) Branch-VNet-JapanEast -> 10.1.0.0/16
2) Branch-VNet-KoreaCentral -> 10.2.0.0/16
3) Core-VNet-WestUS3 -> 10.0.0.0/16
Is this a normal hub-and-spoke or Virtual WAN hub-and-spoke?
Were the peering made as Regional or Global peering?
Any NSG at the NIC or VNET level?
Looking forward to your feedback,
Cheers
This is a normal hub-and-spoke topology, each VNet was created in different regions (Japan East, Korea Central and West US 3) so the peering is global.
There are only 3 NSG at the vNIC level these were created along with the VMs so they have the default rules + SSH inbound allowed.
Understood.
I was taking a look at the peering from the images above but If I am not mistaken this could be set up as follows...
Configure the peering connection in the hub to allow gateway transit.
Configure the peering connection in each spoke to use remote gateways.
Configure all peering connections to allow forwarded traffic.
Also, the UDR routing table needs to have next-hop as Vnet-peering as well.(If I am not mistaken)
Looking forward to your feedback,
Cheers