Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
4,080 questions
Tokens not getting cached in azure OpenAI.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 66%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Sharad Roy
0
Reputation points
When using gpt-4o model in azure OpenAI, cached tokesn count in usage is always coming as zero. The same prompt and history used directly in the openai API gives cached tokens correctly. Do we need to enable any settings to enable caching? I am suing the Openai nodejs sdk for both azure and direct openai.
Azure OpenAI Service
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 88%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator2025-02-27T09:43:12.4033333+00:00 Hi Sharad Roy
It seems you're having trouble with prompt caching in Azure OpenAI Service.
Here are some steps:
- Verify that the model you're utilizing is compatible with prompt caching. In Azure OpenAI, prompt caching is only supported by specific models such as gpt-4o-2024-11-20 and gpt-4o-mini-2024-07-18.
- The prompt must be a minimum of 1,024 tokens in length, and the first 1,024 tokens must be identical for caching to work.
- Prompt caching is enabled by default for supported models,
Kindly refer below link: prompt-caching
Thank You.
-
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator
2025-02-28T04:17:44.25+00:00 -
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator
2025-03-02T10:56:19.53+00:00 Hi Sharad Roy
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
Thank You.
-
Sharad Roy 0 Reputation points
2025-03-02T11:26:48.1066667+00:00 Hi @Saideep Anchuri
Yes, we are using gpt-4o-2024-08-06, and our prompt has more than 2000 tokens. We have tried the same prompt directly on OpenAI api (not through azure), and there caching is working. But through azure we are getting 0 cached tokens. -
Sharad Roy 0 Reputation points
2025-03-02T11:30:19.6733333+00:00 Hi @Saideep Anchuri
We are using gpt-4o-2024-08-06 model and our prompt has more than 2000 tokens. We tried the same prompt directly on OpenAI api (not through azure) and there caching is working. But when trying through azure we are getting 0 cached tokens. -
Saideep Anchuri 9,425 Reputation points Microsoft External Staff Moderator
2025-03-02T11:40:53.89+00:00 Hi Sharad Roy
The gpt-4o-2024-08-06 model does support prompt caching. For prompt caching to work, the request must be at least 1,024 tokens in length, and the first 1,024 tokens in the prompt must be identical. Cache hits will occur for every 128 additional identical tokens after the first 1,024 tokens. A single character difference in the first 1,024 tokens will result in a cache miss, characterized by a cached_tokens value of 0
Kindly refer below link: prompt-caching
Thank You.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 3%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Markus von Staden 25 Reputation points2025-03-02T15:03:39.9166667+00:00 Same problem here. Also using 08-06
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 16%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJD%3C/text%3E%3C/svg%3E)
Jansen Derr 0 Reputation points2025-03-20T21:44:39.21+00:00 Also having this problem using 08-06. The caching feature is bugged. Please have engineering look into this.
-
Jansen Derr 0 Reputation points
2025-03-20T21:45:10.4733333+00:00
Sign in to comment
Sign in to answer