Questions on Azure OpenAI Chatbot capabilities

Question

   Public Async Function ChatBot(<FromBody> body As GetQueryParams) As Task(Of ActionResult)
       Dim rs As New DataSet
       Dim err_no As Long
       Dim identity = CType(User.Identity, ClaimsIdentity)

       Dim userInput As String = body.userQuery

       Dim options As New ChatCompletionOptions()
       options.AddDataSource(New AzureSearchChatDataSource() With {
           .Endpoint = New Uri(searchEndpoint),
           .IndexName = searchIndex,
           .Authentication = DataSourceAuthentication.FromApiKey(searchKey),
           .Strictness = 4,
           .TopNDocuments = 4,
           .InScope = True,
           .RoleInformation = "You are a customer support, request for live agent if you unable to answer the query."
       })

       Dim response As New StringBuilder()
       Dim chatUpdates = chatClient.CompleteChatStreamingAsync(New UserChatMessage() {New UserChatMessage(userInput)}, options)

       Dim enumerator = chatUpdates.GetAsyncEnumerator()
       While Await enumerator.MoveNextAsync()
           Dim chatUpdate = enumerator.Current
           For Each contentPart In chatUpdate.ContentUpdate
               response.Append(contentPart.Text)
           Next
       End While

       Return Ok(response.ToString())
   End Function

The above code is how I call the Azure OpenAI API by creating my own API on my own server. I'm using the chatgpt-4o model and integrated with the Azure AI Search feature using my own data (restricted to my data only). The only thing I need to do is to input more files and information to my Azure Blob Storage (connected to Azure AI Search), so that my model have more wide range of information based on my knowledge files. With that said, it seems that chatgpt-4o model is not a model for fine-tuning, so the only way to improve my model is to have more data and information that my Azure AI Search can be searched through.

1. What are some better solutions for prompt engineering? As when using the "Add Your Data" feature, it actually disabled the examples (some examples to show the chat what responses I want). So if that be it, I just have to worry about adding my own data files to the blob storage right?
2. Instead of removing the referencing (for e.g. [doc4] or [doc1] at the end of the referenced sentences) using regular expression, can I access it without the reference? It will be most convenient if that is possible.
3. What are the capabilities and limitations of Azure OpenAI for creating a customer support chatbot, and what are some of the best solutions when using Azure OpenAI models? Like what I have experienced is the Document Intelligence where it have prebuilt-models, while also Custom Models, I can do a lot for the custom models for training different specific cases in different scenario. What about Azure OpenAI? Like I wanted to trained on the agent being a professional Customer Support using prompt engineering techniques, is that even capable within Azure OpenAI (or specifically to say is it possible to do it with my current situation or I have to choose other models)?
4. How can I include a loop in my API so that it retrieves and posts answers for each query within the loop, instead of calling the API every time a user enters a new query? Is this something that I need to handle on the front-end or can I do it within the API? Is including while loop the solution?

Answer 1

Hi @Tom Chow,

Thank you for reaching out to Microsoft Q&A forum!

To enhance the performance of the chatgpt-4o model, consider enriching your Azure Blob Storage with more relevant data and utilizing detailed prompts to guide the model’s responses. Here is the info for you queries:

• Effective prompt engineering involves crafting detailed prompts and providing examples to ensure the model delivers accurate and contextually relevant responses.
• If your model's responses include unwanted references, you can clean up the text by applying post-processing techniques to remove these references.
• While Azure OpenAI models excel at providing context-specific responses, they cannot be fine-tuned. Instead, focus on prompt engineering and data enrichment to achieve better results.
• To handle multiple queries efficiently, implement looping in your API to process batches of queries or manage them through the front end for a smoother user experience.

Feel free to modify these based on your specific needs!

I hope you understand! Thank you.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful.