Adding System message in the Azure open ai chat completion

Question

Hi Team, By referring the below document, https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions I have added system message in the chat completion while fetching the answer from the private KB. I have updated this system message to handle the private KB no answer case scenario. Is I have added the system message in the below format, Add(new ChatMessage(ChatRole.System, $"You are an AI assistant that helps people find information. - Answer in 50 words or less. Use bullet lists wherever possible.- If you're unsure of an answer, you can say I don't know or I'm not sure and ask user to provide more context on the question.") Asked a question out of private KB and i got the below response

Also in system message I have added , the answer should contain only 50 words or less and in parameter i have configured 100 tokens. In the result which i git contains more than 50 words,

Could anyone help to undetsand the use of system message while using private KB for data fetch.

Answer 1

Hi @lakshmi
Welcome to Microsoft Q&A! Thanks for posting the question.

As stated in the documentation, the system message is not a strict rule that the model has to follow. It is more like a suggestion or a hint that the model can use to improve its output. The model may still generate answers that are longer than 50 words or do not use bullet lists, depending on the question and the private KB content.

Both the system message and the token limit are two different ways to control the chat completion model output. The system message is a soft constraint that guides the model to generate answers that match your expectations. The token limit is a hard constraint that limits the model to generate answers that fit within the token budget. But neither of them guarantees that the model will generate answers that are exactly 50 words or less. If you want to enforce a strict word limit, you may need to post-process the model output and truncate it if it exceeds the desired length. Hope this helps.

Thanks Saurabh