Teams Toolkit ChatBot - Streaming Response from LLM

Question

We've been experimenting with MS Teams Toolkit to build a chatbot using Python and successfully deployed a basic bot that interacts with an LLM. However, we've noticed that the bot waits until the entire response is received before displaying it in the UI, which may not provide the best user experience.

What is the recommended approach to enable streaming responses in the Teams bot chat UI? We'd like to display incoming messages in chunks to keep users engaged while the LLM continues generating and streaming text in the background, as most LLMs natively support this capability.

Any guidance or best practices to achieve this while using Teams Toolkit SDK would be greatly appreciated!

Answer 1

To enable streaming responses in the Teams bot chat UI using the Teams Toolkit SDK, particularly with Python, you can follow these steps to configure your AI-powered bot to stream messages. This approach will allow the bot to send chunks of messages as they are generated, providing a more engaging experience for the user.

1. Enable Streaming for AI-powered Bot:
   • Use the DefaultAugmentation class in the config.json file of your bot.
   • Set the stream parameter to true in the OpenAIModel declaration within your main application file (bot.py).
2. Set Informative Messages:
   • Define an informative message that will appear to the user every time the bot sends an update.
3. Format the Final Streamed Message:
   • Configure the bot to format text messages and simple markdown while they are being streamed.
   • Ensure that attachments like Adaptive Cards are only sent once the final message chunk is complete.

Ref: https://learn.microsoft.com/en-us/microsoftteams/platform/bots/streaming-ux?tabs=csharp#configure-streaming-bot-messages

Thanks, 

Prasad Das

************************************************************************* 

If the response is helpful, please click "Accept Answer" and upvote it. You can share your feedback via Microsoft Teams Developer Feedback link. Click here to escalate.