Azure Function Streaming Response

Question

Is it possible to stream a response from an Azure Function written in Python?

According to some articles I found, it's not possible, but I was hopeful when I saw some sources implementing it using FastAPI. Since Functions support FastAPI I build an endpoint using this snippet on streaming langchain responses. It works locally when I run the FastAPI server directly using uvicorn. I can also deploy the function app and the endpoint answers requests.

However, the response is never "actually" streamed. It loads until the full stream has been received and only then am I able to display the data. That's obviously not the point and I would like confirmation that Azure Functions simply do not support streaming of any kind.

If that's the case: Please Microsoft, add this feature. You are the frontrunner on Copilots and enterprise LLMs, yet your serverless compute does not support streaming.

Answer 1

Hello - You may be seeing so because the Python worker process runs behind the host process. I'll relay the feedback to the Functions team and get a confirmation.

UPDATE 11/14/23:

The product team will be releasing a new version of the Python worker that'll support streaming req/res. Please track the following discussion: https://github.com/Azure/azure-functions-python-worker/discussions/1349#discussioncomment-7569591

In the meantime, it's reported that the SubApplication pattern works (as discussed in this thread) but I believe the timeouts will still be a potential issue.

---

Please "Accept Answer" if the answer is helpful so that others in the community may benefit from your experience.