The topoplogy is spoke subnet ---> Aure LB ---> 2x Palo VM firewalls -> express route --> on-prem Palo firewall --> on-prem server user at spok subnet send files to onprem is very slow. we did iperf test from a subnet in the spoke vnet to an onprem test server. There are drops on both of the firewalls that behind the LB. The dropped packets are normal tcp ack, fin-ack, rst ack cwr, and tcp retrsnmission. we did another iperf test from a different subnet in the same spoke vnet and skip the Azure LB , just go through one of the Palo vm firewall. Then there is no drops on this Palo firewall. also, there is no drop on the on-prem palo firewall. what could cause the drop on the palo vm firewalls when behind the Azure LB? could anyone help? Thank you!

@Vanessa Xu , Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well. Can you please confirm how exactly are you identifying there is a packet drop at the Azure Palo Alto Firewall NVAs? Are you saying the packets never reached the Azure Palo Alto Firewall NVAs? Or they are reaching the Azure Palo Alto Firewall NVAs and are dropped after that. Cheers, Kapil

Palo VM firewall drop packets behind Azure load balancer

Accepted answer

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-04T04:34:06.3633333+00:00
@Vanessa Xu ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Can you please confirm how exactly are you identifying there is a packet drop at the Azure Palo Alto Firewall NVAs?

Are you saying the packets never reached the Azure Palo Alto Firewall NVAs?

Or they are reaching the Azure Palo Alto Firewall NVAs and are dropped after that.

Cheers,

Kapil
Please sign in to rate this answer.
Vanessa Xu 20 Reputation points

2024-07-04T17:04:30.8466667+00:00

we did pacap on firewall during the iperf tests and the dropping happens on the firewall.

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-05T14:58:43.8466667+00:00

@Vanessa Xu ,

Can you be more specific?

Is it the NVAs (Azure VMs acting as Firewall) or the OnPrem firewall?

Vanessa Xu 20 Reputation points

2024-07-08T13:39:08.97+00:00

the Azure palo firewalls are vms. the on-prem firewalls are hardware and no drops during test.

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-08T14:18:39.0466667+00:00

@Vanessa Xu ,

In that case,

Are you seeing the packets logged in the NVAs (Palo Firewall VMs) ?
OR

Are you not at all seeing the packets entering the NVAs?

Cheers,

Kapil

Vanessa Xu 20 Reputation points

2024-07-08T15:43:50.1866667+00:00

we did pcap on the palo vm on Azure and see drops.

How could we do packet capture on Azure before it enter the NVAs?

The Azure LB is using the default session perssitence setting. Does other methods helps (i.e. client)?

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-08T17:24:13.2266667+00:00

@Vanessa Xu ,

You cannot do packet capture at Load Balancer.

If you are able to see the packets entering the NVA and you are mentioning that the packets are getting dropped here,

It could also mean the NVA OS is dropping the packets? (instead of Platform dropping it)

When you say "see drops",

do you mean the NVA is flagging the packet as invalid and dropping it?

Or simply you do not see this packet entering the next hop?

As next steps,

You could set Session persistence as Client IP & protocol and give it a try

But this mainly depends on your NVA's support for active/active flows

Is this a new setup?

Or was this an old set up and the issue recently showed up?

If so, did you make any recent configurational changes that resulted in this?

Cheers,

Kapil

Vanessa Xu 20 Reputation points

2024-07-31T13:16:45.7766667+00:00

Hi Kapil,

the packet drop was cappurted on the NVA (palo firewall VMs). However, Palo doesn't know in which stage these packets were dropped.

Later, we changed the LB persistent method from default (5 tuple) to client and protocol (3 tuple), there is no more dropping packets at the NVA. Also the vnet to onprem file share and transfer performance got improved a lot.

We just leave this as is as we the further debug on the NVA will require a downtime and we don't have time do perform this.

Thank you so much for your help!

Vanessa

KapilAnanth-MSFT 47,046 Reputation points Microsoft Employee

2024-07-31T13:28:11.5633333+00:00

@Vanessa Xu ,

I'm glad that the issue resolved after switching Session persistence to Client IP & protocol and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll summarize our discussion.

Please "Accept" the answer as original posters help the community find answers faster by identifying the correct answer.

Thanks for your continued contribution on Q&A and appreciate much for taking the time to share your feedback.

Cheers,

Kapil
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Palo VM firewall drop packets behind Azure load balancer

0 additional answers

Your answer