Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
In this article, you use Python or .NET to send chat completion requests, build a multi-turn conversation, and manage the conversation's token budget.
Chat models are language models optimized for conversational interfaces. Unlike older text-in and text-out completion models, chat models accept a transcript of messages and return a model-generated message. This format supports multi-turn conversations and nonchat scenarios.
Use the message format described in this article instead of prompting chat models like older completion models. Otherwise, the models might produce verbose or less useful responses.
Tip
For new apps, consider building on the Responses API instead of Chat Completions. To upgrade an existing app, see Azure OpenAI To Responses.
Prerequisites
- An Azure OpenAI resource with a chat completion model deployment. To create a resource and deploy a model, see Create a resource and deploy a model with Azure OpenAI.
- Install the OpenAI Python library:
pip install openai. - For Microsoft Entra ID authentication, install Azure Identity (
pip install azure-identity) and the Azure CLI. Assign theCognitive Services Userrole to your user account, and then runaz login. - For the token-counting example, install tiktoken:
pip install tiktoken. - If you use API keys, set the
AZURE_OPENAI_API_KEYenvironment variable.
- The .NET 8.0 SDK or later.
- For Microsoft Entra ID authentication, install the Azure CLI and assign the
Cognitive Services Userrole to your user account. - If you use API keys, set the
AZURE_OPENAI_API_KEYenvironment variable.
In the code samples, replace YOUR-RESOURCE-NAME with your Azure OpenAI resource name and YOUR-DEPLOYMENT-NAME with your model deployment name.
Set up
Save each complete example as chat.py, and then run it with python chat.py.
Work with chat completion models
The following code snippet shows the most basic way to interact with models that use the Chat Completions API.
Note
The Responses API uses the same chat style of interaction, but supports the latest features that aren't available with the older Chat Completions API.
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(
DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
api_key=token_provider,
)
response = client.chat.completions.create(
model="YOUR-DEPLOYMENT-NAME", # Replace with your model deployment name.
messages=[
{"role": "system", "content": "Assistant is a large language model trained by OpenAI."},
{"role": "user", "content": "Who were the founders of Microsoft?"}
]
)
#print(response)
print(response.model_dump_json(indent=2))
print(response.choices[0].message.content)
{
"id": "chatcmpl-8GHoQAJ3zN2DJYqOFiVysrMQJfe1P",
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"content": "Microsoft was founded by Bill Gates and Paul Allen. They established the company on April 4, 1975. Bill Gates served as the CEO of Microsoft until 2000 and later as Chairman and Chief Software Architect until his retirement in 2008, while Paul Allen left the company in 1983 but remained on the board of directors until 2000.",
"role": "assistant"
},
"content_filter_results": {
"hate": {
"filtered": false,
"severity": "safe"
},
"self_harm": {
"filtered": false,
"severity": "safe"
},
"sexual": {
"filtered": false,
"severity": "safe"
},
"violence": {
"filtered": false,
"severity": "safe"
}
}
}
],
"created": 1698892410,
"model": "gpt-4o",
"object": "chat.completion",
"usage": {
"completion_tokens": 73,
"prompt_tokens": 29,
"total_tokens": 102
},
"prompt_filter_results": [
{
"prompt_index": 0,
"content_filter_results": {
"hate": {
"filtered": false,
"severity": "safe"
},
"self_harm": {
"filtered": false,
"severity": "safe"
},
"sexual": {
"filtered": false,
"severity": "safe"
},
"violence": {
"filtered": false,
"severity": "safe"
}
}
}
]
}
Microsoft was founded by Bill Gates and Paul Allen. They established the company on April 4, 1975. Bill Gates served as the CEO of Microsoft until 2000 and later as Chairman and Chief Software Architect until his retirement in 2008, while Paul Allen left the company in 1983 but remained on the board of directors until 2000.
Every response includes finish_reason. The possible values for finish_reason are:
- stop: API returned complete model output.
- length: Incomplete model output because of the
max_completion_tokensparameter or the token limit. - content_filter: Omitted content because of a content filter flag.
- tool_calls: The model called a tool.
- function_call: The model called a function. This value is deprecated.
For streaming responses, finish_reason is null until the final chunk completes the response.
Set max_completion_tokens high enough for the expected response. A higher value helps prevent the model from stopping before it reaches the end of the message.
Work with the Chat Completions API
OpenAI trained chat completion models to accept input formatted as a conversation. The messages parameter takes an array of message objects with a conversation organized by role. When you use the Python API, a list of dictionaries is used.
The format of a basic chat completion is:
messages = [
{"role": "system", "content": "Provide context or instructions to the model."},
{"role": "user", "content": "The user's message goes here."},
]
A conversation with one example answer followed by a question would look like:
messages = [
{"role": "system", "content": "Provide context or instructions to the model."},
{"role": "user", "content": "Example question goes here."},
{"role": "assistant", "content": "Example answer goes here."},
{"role": "user", "content": "First question for the model to answer."},
]
System role
The system role, also known as the system message, is included at the beginning of the array. This message provides the initial instructions to the model. You can provide various information in the system role, such as:
- A brief description of the assistant.
- Personality traits of the assistant.
- Instructions or rules you want the assistant to follow.
- Data or information needed for the model, such as relevant questions from an FAQ.
Customize the system role for your use case or include basic instructions. The system message is optional, but include at least a basic one to get the best results.
Messages
After the system role, you can include a series of messages between the user and the assistant.
message = {"role": "user", "content": "What is thermodynamics?"}
To trigger a response from the model, end with a user message to indicate that it's the assistant's turn to respond. You can also include a series of example messages between the user and the assistant as a way to do few-shot learning.
Message prompt examples
The following section shows examples of different styles of prompts that you can use with chat completions models. These examples are only a starting point. You can experiment with different prompts to customize the behavior for your own use cases.
Basic example
If you want your chat completions model to behave similarly to chatgpt.com, you can use a basic system message like Assistant is a large language model trained by OpenAI.
messages = [
{"role": "system", "content": "Assistant is a large language model trained by OpenAI."},
{"role": "user", "content": "Who were the founders of Microsoft?"},
]
Example with instructions
For some scenarios, you might want to give more instructions to the model to define guardrails for what the model is able to do.
messages = [
{"role": "system", "content": """Assistant is an intelligent chatbot designed to help users answer tax-related questions.
Instructions:
- Only answer questions related to taxes.
- If you're unsure of an answer, say "I don't know" or "I'm not sure" and recommend the IRS website."""},
{"role": "user", "content": "When are my taxes due?"},
]
Use data for grounding
You can also include relevant data or information in the system message to give the model extra context for the conversation. If you need to include only a small amount of information, you can hard code it in the system message. If you have a large amount of data that the model should be aware of, you can use embeddings or a product like Azure AI Search to retrieve the most relevant information at query time.
messages = [
{"role": "system", "content": """Assistant helps users answer technical questions about Azure OpenAI in Microsoft Foundry Models. Only answer questions using the following context. If the context doesn't contain the answer, say 'I don't know.'
Context:
- Azure OpenAI provides REST API access to OpenAI models, including GPT-5, GPT-4.1, and Embeddings model series.
- Azure OpenAI gives customers advanced language AI with GPT-5, GPT-image, and Embeddings models with the security and enterprise capabilities of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition between the services.
- At Microsoft, we're committed to advancing AI according to principles that put people first."""},
{"role": "user", "content": "What is Azure OpenAI?"},
]
Few-shot learning with chat completion
You can also give few-shot examples to the model. The approach for few-shot learning has changed slightly because of the new prompt format. You can now include a series of messages between the user and the assistant in the prompt as few-shot examples. By using these examples, you can seed answers to common questions to prime the model or teach particular behaviors to the model.
This example shows how to use few-shot learning with current chat completion models such as gpt-5-mini and gpt-5. Experiment with different approaches to find what works best for your use case.
messages = [
{"role": "system", "content": "Assistant helps users answer tax-related questions."},
{"role": "user", "content": "When do I need to file my taxes by?"},
{"role": "assistant", "content": "Check the current individual filing deadline at https://www.irs.gov/filing/individuals/when-to-file."},
{"role": "user", "content": "How can I check the status of my tax refund?"},
{"role": "assistant", "content": "Check your refund status at https://www.irs.gov/refunds."},
]
Use chat completion for nonchat scenarios
The Chat Completions API is designed to work with multi-turn conversations, but it also works well for nonchat scenarios.
For example, for an entity extraction scenario, you might use the following prompt:
messages = [
{"role": "system", "content": """You extract entities from text and return them as a JSON object with this format:
{
"name": "",
"company": "",
"phone_number": ""
}"""},
{"role": "user", "content": "Hello. My name is Robert Smith. I'm calling from Contoso Insurance, Delaware. My colleague mentioned that you are interested in learning about our comprehensive benefits policy. Could you give me a call back at (555) 346-9322 when you get a chance so we can go over the benefits?"},
]
Create a basic conversation loop
The preceding examples show the basic mechanics of interacting with the Chat Completions API. This example shows how to create a conversation loop that performs the following actions:
- Continuously takes console input and properly formats it as part of the messages list as user role content.
- Outputs responses that are printed to the console and formatted and added to the messages list as assistant role content.
Every time you ask a new question, the request sends the running conversation transcript along with the latest question. Because the model has no memory, send an updated transcript with each question or the model loses the context of previous questions and answers.
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(
DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
api_key=token_provider,
)
conversation = [{"role": "system", "content": "You are a helpful assistant."}]
while True:
user_input = input("Q:")
conversation.append({"role": "user", "content": user_input})
response = client.chat.completions.create(
model="YOUR-DEPLOYMENT-NAME", # Replace with your model deployment name.
messages=conversation
)
conversation.append({"role": "assistant", "content": response.choices[0].message.content})
print("\n" + response.choices[0].message.content + "\n")
When you run the preceding code, you get a blank console window. Enter your first question in the window and then select the Enter key. After the response is returned, you can repeat the process and keep asking questions.
Manage conversations
The previous example runs until you hit the model's token limit (context window). With each question asked and answer received, the messages list grows in size. The combined token count of your messages plus the requested output tokens must stay within the model's limit, or the request fails. Consult the models page for current token limits.
It's your responsibility to ensure that the prompt and completion fall within the token limit. For longer conversations, you need to keep track of the token count and only send the model a prompt that falls within the limit. Alternatively, with the responses API you can have the API handle truncation/management of the conversation history for you.
The following code sample uses OpenAI's tiktoken library to trim a conversation at a 4,096-token demonstration threshold. Set token_limit to the context window of your deployed model for production use.
You might need to upgrade tiktoken by running pip install --upgrade tiktoken.
import tiktoken
from openai import OpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
token_provider = get_bearer_token_provider(
DefaultAzureCredential(), "https://ai.azure.com/.default"
)
client = OpenAI(
base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/",
api_key=token_provider,
)
system_message = {"role": "system", "content": "You are a helpful assistant."}
max_response_tokens = 250
token_limit = 4096
conversation = []
conversation.append(system_message)
def num_tokens_from_messages(messages, model="gpt-4o"):
"""Return the number of tokens used by a list of messages."""
try:
encoding = tiktoken.encoding_for_model(model)
except KeyError:
print("Warning: model not found. Using o200k_base encoding.")
encoding = tiktoken.get_encoding("o200k_base")
if model in {
"gpt-4o",
"gpt-4o-mini",
"gpt-5",
"gpt-4.1",
"o1",
"o1-mini",
"o3",
"o3-mini",
"o4-mini",
}:
tokens_per_message = 3
tokens_per_name = 1
elif any(model.startswith(prefix) for prefix in [
"gpt-4o-",
"gpt-5-",
"gpt-4.1-",
"o1-",
"o3-",
"o4-mini-",
]):
tokens_per_message = 3
tokens_per_name = 1
else:
raise NotImplementedError(
f"""num_tokens_from_messages() is not implemented for model {model}. """
)
num_tokens = 0
for message in messages:
num_tokens += tokens_per_message
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if key == "name":
num_tokens += tokens_per_name
num_tokens += 3
return num_tokens
while True:
user_input = input("Q:")
conversation.append({"role": "user", "content": user_input})
conv_history_tokens = num_tokens_from_messages(conversation, model="gpt-4o")
while conv_history_tokens + max_response_tokens >= token_limit:
del conversation[1]
conv_history_tokens = num_tokens_from_messages(conversation, model="gpt-4o")
response = client.chat.completions.create(
model="YOUR-DEPLOYMENT-NAME",
messages=conversation,
temperature=0.7,
max_completion_tokens=max_response_tokens
)
conversation.append({"role": "assistant", "content": response.choices[0].message.content})
print("\n" + response.choices[0].message.content + "\n")
In this example, after the token count is reached, the oldest messages in the conversation transcript are removed. For efficiency, del is used instead of pop(). We start at index 1 to always preserve the system message and only remove user or assistant messages. Over time, this method of managing the conversation can cause the conversation quality to degrade as the model gradually loses the context of the earlier portions of the conversation.
An alternative approach is to limit the conversation duration to the maximum token length or a specific number of turns. After the maximum token limit is reached, the model would lose context if you were to allow the conversation to continue. You can prompt the user to begin a new conversation and clear the messages list to start a new conversation with the full token limit available.
The token counting portion of the code demonstrated previously is a simplified version of one of OpenAI's cookbook examples.
Troubleshooting
Failed to create completion as the model generated invalid Unicode output
- Error code: 500
- Error message:
500 - InternalServerError: Error code: 500 - {"error": {"message": "Failed to create completion as the model generated invalid Unicode output"}} - Workaround: Reduce the prompt temperature to less than 1, and use a client with retry logic. Retrying the request often succeeds.
Common errors
- 401/403 (authentication): Verify your API key, or confirm you have Microsoft Entra ID access to the Azure OpenAI resource.
- 400/404 (deployment not found): Confirm that
modelmatches your deployment name. - Invalid URL: Confirm that
base_urlends with/openai/v1/.
Set up
Create a new .NET console application:
dotnet new console -n chat-completions cd chat-completionsInstall the required NuGet packages:
dotnet add package OpenAI dotnet add package Azure.IdentityThe OpenAI package is stable. The Microsoft Entra ID examples use an experimental custom authentication constructor and suppress the
OPENAI001warning.For keyless authentication with Microsoft Entra ID, sign in to Azure:
az login
Work with chat completion models
The following code snippet shows the most basic way to interact with models that use the Chat Completions API.
Note
The responses API uses the same chat style of interaction, but supports the latest features that aren't supported with the older chat completions API.
using Azure.Identity;
using OpenAI;
using OpenAI.Chat;
using System.ClientModel.Primitives;
#pragma warning disable OPENAI001
BearerTokenPolicy tokenPolicy = new(
new DefaultAzureCredential(),
"https://ai.azure.com/.default");
ChatClient client = new(
model: "YOUR-DEPLOYMENT-NAME",
authenticationPolicy: tokenPolicy,
options: new OpenAIClientOptions()
{
Endpoint = new Uri("https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/")
}
);
ChatCompletion completion = await client.CompleteChatAsync(
[
new SystemChatMessage("Assistant is a large language model trained by OpenAI."),
new UserChatMessage("Who were the founders of Microsoft?"),
]);
Console.WriteLine(completion.Content[0].Text);
Microsoft was founded by Bill Gates and Paul Allen. They established the company on April 4, 1975. Bill Gates served as the CEO of Microsoft until 2000 and later as Chairman and Chief Software Architect until his retirement in 2008, while Paul Allen left the company in 1983 but remained on the board of directors until 2000.
Every response includes a FinishReason. The possible values for FinishReason are:
- Stop: The API returned complete model output.
- Length: Incomplete model output because of the
MaxOutputTokenCountparameter or the token limit. - ContentFilter: Omitted content because of a content filter flag.
- ToolCalls: The model called a tool.
- FunctionCall: The model called a function. This value is deprecated.
Set MaxOutputTokenCount high enough for the expected response. A higher value helps prevent the model from stopping before it reaches the end of the message.
Work with the Chat Completions API
OpenAI trained chat completion models to accept input formatted as a conversation. The messages parameter takes an array of message objects with a conversation organized by role. When you use the .NET SDK, you use strongly typed message classes for each role.
The format of a basic chat completion is:
new SystemChatMessage("Provide some context and/or instructions to the model"),
new UserChatMessage("The user's message goes here")
A conversation with one example answer followed by a question would look like:
new SystemChatMessage("Provide some context and/or instructions to the model."),
new UserChatMessage("Example question goes here."),
new AssistantChatMessage("Example answer goes here."),
new UserChatMessage("First question/message for the model to actually respond to.")
System role
The system role, also known as the system message, is included at the beginning of the array. This message provides the initial instructions to the model. You can provide various information in the system role, such as:
- A brief description of the assistant.
- Personality traits of the assistant.
- Instructions or rules you want the assistant to follow.
- Data or information needed for the model, such as relevant questions from an FAQ.
Customize the system role for your use case or include basic instructions. The system message is optional, but include at least a basic one to get the best results.
Messages
After the system role, you can include a series of messages between the user and the assistant.
new UserChatMessage("What is thermodynamics?")
To trigger a response from the model, end with a user message to indicate that it's the assistant's turn to respond. You can also include a series of example messages between the user and the assistant as a way to do few-shot learning.
Message prompt examples
The following section shows examples of different styles of prompts that you can use with chat completions models. These examples are only a starting point. You can experiment with different prompts to customize the behavior for your own use cases.
Basic example
If you want your chat completions model to behave similarly to chatgpt.com, you can use a basic system message like Assistant is a large language model trained by OpenAI.
new SystemChatMessage("Assistant is a large language model trained by OpenAI."),
new UserChatMessage("Who were the founders of Microsoft?")
Example with instructions
For some scenarios, you might want to give more instructions to the model to define guardrails for what the model is able to do.
new SystemChatMessage(@"Assistant is an intelligent chatbot designed to help users answer their tax related questions.
Instructions:
- Only answer questions related to taxes.
- If you're unsure of an answer, you can say ""I don't know"" or ""I'm not sure"" and recommend users go to the IRS website for more information."),
new UserChatMessage("When are my taxes due?")
Use data for grounding
You can also include relevant data or information in the system message to give the model extra context for the conversation. If you need to include only a small amount of information, you can hard code it in the system message. If you have a large amount of data that the model should be aware of, you can use embeddings or a product like Azure AI Search to retrieve the most relevant information at query time.
new SystemChatMessage(@"Assistant is an intelligent chatbot designed to help users answer technical questions about Azure OpenAI in Microsoft Foundry Models. Only answer questions using the context below and if you're not sure of an answer, you can say 'I don't know'.
Context:
- Azure OpenAI provides REST API access to OpenAI models, including GPT-5, GPT-4.1, and Embeddings model series.
- Azure OpenAI gives customers advanced language AI with GPT-5, GPT-image, and Embeddings models with the security and enterprise capabilities of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition between the services.
- At Microsoft, we're committed to the advancement of AI driven by principles that put people first. Microsoft has made significant investments to help guard against abuse and unintended harm, which includes requiring applicants to show well-defined use cases, incorporating Microsoft's principles for responsible AI use."),
new UserChatMessage("What is Azure OpenAI?")
Few-shot learning with chat completion
You can also give few-shot examples to the model. You can include a series of messages between the user and the assistant in the prompt as few-shot examples. By using these examples, you can seed answers to common questions to prime the model or teach particular behaviors to the model.
This example uses current chat completion models such as gpt-5-mini and gpt-5.
new SystemChatMessage("Assistant is an intelligent chatbot designed to help users answer their tax related questions."),
new UserChatMessage("When do I need to file my taxes by?"),
new AssistantChatMessage("Check the current individual filing deadline at https://www.irs.gov/filing/individuals/when-to-file."),
new UserChatMessage("How can I check the status of my tax refund?"),
new AssistantChatMessage("Check your refund status at https://www.irs.gov/refunds.")
Use chat completion for nonchat scenarios
The Chat Completions API is designed to work with multi-turn conversations, but it also works well for nonchat scenarios.
For example, for an entity extraction scenario, you might use the following prompt:
new SystemChatMessage(@"You are an assistant designed to extract entities from text. Users will paste in a string of text and you will respond with entities you've extracted from the text as a JSON object. Here's an example of your output format:
{
""name"": """",
""company"": """",
""phone_number"": """"
}"),
new UserChatMessage("Hello. My name is Robert Smith. I'm calling from Contoso Insurance, Delaware. My colleague mentioned that you are interested in learning about our comprehensive benefits policy. Could you give me a call back at (555) 346-9322 when you get a chance so we can go over the benefits?")
Create a basic conversation loop
The preceding examples show the basic mechanics of interacting with the Chat Completions API. This example shows how to create a conversation loop that performs the following actions:
- Continuously takes console input and properly formats it as part of the messages list as user role content.
- Outputs responses that are printed to the console and formatted and added to the messages list as assistant role content.
Every time you ask a new question, the request sends the running conversation transcript along with the latest question. Because the model has no memory, send an updated transcript with each question or the model loses the context of previous questions and answers.
using Azure.Identity;
using OpenAI;
using OpenAI.Chat;
using System.ClientModel.Primitives;
#pragma warning disable OPENAI001
BearerTokenPolicy tokenPolicy = new(
new DefaultAzureCredential(),
"https://ai.azure.com/.default");
ChatClient client = new(
model: "YOUR-DEPLOYMENT-NAME",
authenticationPolicy: tokenPolicy,
options: new OpenAIClientOptions()
{
Endpoint = new Uri("https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/")
}
);
List<ChatMessage> conversation =
[
new SystemChatMessage("You are a helpful assistant."),
];
while (true)
{
Console.Write("Q: ");
string? userInput = Console.ReadLine();
if (string.IsNullOrWhiteSpace(userInput)) break;
conversation.Add(new UserChatMessage(userInput));
ChatCompletion response = await client.CompleteChatAsync(conversation);
string assistantMessage = response.Content[0].Text;
conversation.Add(new AssistantChatMessage(assistantMessage));
Console.WriteLine($"\n{assistantMessage}\n");
}
When you run the preceding code, you get a blank console window. Enter your first question in the window and then select the Enter key. After the response is returned, you can repeat the process and keep asking questions.
Manage conversations
The previous example runs until the model's token limit (context window) is reached. With each question asked and answer received, the conversation list grows in size. The combined token count of your messages plus the requested output tokens must stay within the model's limit, or the request fails. Consult the models page for current token limits.
It's your responsibility to ensure that the prompt and completion fall within the token limit. For longer conversations, you need to keep track of the token count and only send the model a prompt that falls within the limit. Alternatively, with the responses API you can have the API handle truncation and management of the conversation history for you.
The following code sample trims the conversation at a 4,096-token demonstration threshold. Set TokenLimit to the context window of your deployed model for production use. The sample removes the oldest non-system messages to keep the conversation within bounds.
Install the Microsoft.ML.Tokenizers and Microsoft.ML.Tokenizers.Data.O200kBase packages for accurate token counting:
dotnet add package Microsoft.ML.Tokenizers
dotnet add package Microsoft.ML.Tokenizers.Data.O200kBase
using Azure.Identity;
using Microsoft.ML.Tokenizers;
using OpenAI;
using OpenAI.Chat;
using System.ClientModel.Primitives;
#pragma warning disable OPENAI001
BearerTokenPolicy tokenPolicy = new(
new DefaultAzureCredential(),
"https://ai.azure.com/.default");
ChatClient client = new(
model: "YOUR-DEPLOYMENT-NAME",
authenticationPolicy: tokenPolicy,
options: new OpenAIClientOptions()
{
Endpoint = new Uri("https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/")
}
);
const int MaxResponseTokens = 250;
const int TokenLimit = 4096;
var tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o");
List<ChatMessage> conversation =
[
new SystemChatMessage("You are a helpful assistant."),
];
static int CountTokens(TiktokenTokenizer tokenizer, IEnumerable<ChatMessage> messages)
{
int count = 3; // base overhead for reply priming
foreach (var message in messages)
{
count += 4; // per-message overhead
string content = message switch
{
SystemChatMessage s => s.Content[0].Text ?? string.Empty,
UserChatMessage u => u.Content[0].Text ?? string.Empty,
AssistantChatMessage a => a.Content[0].Text ?? string.Empty,
_ => string.Empty
};
count += tokenizer.CountTokens(content);
}
return count;
}
while (true)
{
Console.Write("Q: ");
string? userInput = Console.ReadLine();
if (string.IsNullOrWhiteSpace(userInput)) break;
conversation.Add(new UserChatMessage(userInput));
int historyTokens = CountTokens(tokenizer, conversation);
while (historyTokens + MaxResponseTokens >= TokenLimit && conversation.Count > 2)
{
conversation.RemoveAt(1); // remove oldest non-system message
historyTokens = CountTokens(tokenizer, conversation);
}
ChatCompletionOptions options = new() { MaxOutputTokenCount = MaxResponseTokens };
ChatCompletion response = await client.CompleteChatAsync(conversation, options);
string assistantMessage = response.Content[0].Text;
conversation.Add(new AssistantChatMessage(assistantMessage));
Console.WriteLine($"\n{assistantMessage}\n");
}
In this example, after the token count is reached, the oldest messages in the conversation transcript are removed. We always preserve the system message and only remove user or assistant messages. Over time, this method of managing the conversation can cause the conversation quality to degrade as the model gradually loses the context of the earlier portions of the conversation.
An alternative approach is to limit the conversation duration to the maximum token length or a specific number of turns. After the maximum token limit is reached, the model would lose context if you were to allow the conversation to continue. You can prompt the user to begin a new conversation and clear the messages list to start a new conversation with the full token limit available.
Troubleshooting
Failed to create completion as the model generated invalid Unicode output
- Error code: 500
- Error message:
500 - InternalServerError: Error code: 500 - {"error": {"message": "Failed to create completion as the model generated invalid Unicode output"}} - Workaround: Set
TemperatureinChatCompletionOptionsto less than 1, and use a client with retry logic. Retrying the request often succeeds.
Common errors
- 401/403 (authentication): Verify your API key or confirm you have Microsoft Entra ID access to the Azure OpenAI resource.
- 400/404 (deployment not found): Confirm that the model name passed to the
ChatClientconstructor matches your deployment name. - Invalid endpoint: Confirm that the
EndpointURI inOpenAIClientOptionspoints tohttps://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/.