How to enable Streaming response for an App deployed on AKS?

Question

Hi folks,

Context:

We have developed an App written FastAPI, which generates streaming response as the response is bigger in size. We have deployed this App in Azure Kubernetes Service. Application Gateway is configured as well for handling request and responses.

Problem:

We expect the streaming response to come on the front-end. However, we get the entire response at once. And this impacts the user experience.

What we have tried so far:

• We have added below properties in ingress.yml file for the app deployed on AKS:

nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
proxy_set_header: Content-Type text/event-stream
nginx.ingress.kubernetes.io/configuration-snippet: |
  chunked_transfer_encoding on;

• Deployed the app on the VM. This gives me streaming response as expected as no Nginx/App Gateway are involved.
• I ran cURL command from within a pod, and here as well I was getting streaming response.

One thing is clear - the issue is at the Networking/Application Gateway level which is buffering the entire response and sending it at once. Could you pls let me know if I am missing out on anything or any configuration which needs to be enabled? Your help will be really appreciated.

Thanks,

Surya

Answer 1

Hello @Surya Tiwari

I understand that your the issue stems from buffering occurring somewhere in your network layer

Lets try some approaches

1. Application Gateway Settings:
   • Disabling Response Buffering:
   • In your Application Gateway settings, under the "Backend HTTP Settings," explicitly disable response buffering. This should prevent the gateway from accumulating the entire response before forwarding it.
   • Ensure you've configured the same backend pool for the HTTP listener and backend HTTP settings.
   • Connection Draining: If the backend application supports it, consider enabling connection draining in the Application Gateway. This allows in-flight requests to complete before new ones are sent to the backend, potentially minimizing buffering issues.
   • WebSockets Support: If your application relies on persistent connections or WebSockets for real-time communication, ensure that Application Gateway has WebSocket support enabled.
2. Adjusting Ingress Configuration:
   • Explicit Proxy Settings: Consider adding explicit proxy buffering settings in your Ingress configuration to further fine-tune the behavior:

nginx.ingress.kubernetes.io/proxy-buffer-size: "0"

nginx.ingress.kubernetes.io/proxy-buffers: "0"

Also if nothing brings results please try getting some Logs or Even Packet Captures - nmap -

==

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards