TPS drop when connecting to AWS hosted API from Azure

Question

We have API Gateway hosted in AWS EKS and upstream API aka backend hosted in AWS. Also same API Gateway hosted in Azure (AKS).

When we run a perf test on following scenarios
Scenario - 1

• apigateway-us-east-2.aws.com ---> foo-us-east-2.aws.com

Scenario - 2

• apigateway-centralus.azure.com ---> foo-us-east-2.aws.com

Scenario 2 is yielding us 20% drop in TPS compared to scenario 1 and around 1% traffic failed due to connection errors to aws upstream.

Can anyone help me understand what could be potential reason ?

Answer 1

Hello @Ghanapuram, Chaitanya

Thank you for reaching out to the Microsoft Q&A platform.

Based on the provided information, it seems like you are experiencing a drop in TPS and connection errors when using the Azure API Gateway to connect to the AWS upstream API. There could be multiple factors that can affect connectivity success, such as firewall or other traffic management components at the destination, API rate limiting imposed by the destination side, and volumetric DDoS mitigations or transport layer traffic shaping.

To diagnose connection issues, you can use NAT gateway metrics in Azure monitor. Look at packet count at the source and the destination (if available) to determine how many connection attempts were made. Look at dropped packets to see how many packets were dropped by NAT gateway. You can also check for SNAT exhaustion due to NAT gateway configuration, validate connectivity to an endpoint in the same region or elsewhere for comparison, and explore if reducing the rate reduces the occurrence of failures.

It is also important to note that outbound Passive FTP may not work for NAT gateway with multiple public IP addresses, depending on your FTP server configuration. To prevent possible passive FTP connection failures, make sure to check that your NAT gateway is attached to a single public IP address rather than multiple IP addresses or a prefix, and make sure that the passive port range from your NAT gateway is allowed to pass any firewalls that may be at the destination endpoint.

If your investigation is inconclusive, you can open an azure technical support case for further troubleshooting and collect the necessary information for a quicker resolution.