Prompt caching in Azure OpenAI?

Christian 100

OpenAi recently announced Prompt Caching in the API. Apparently it caches input tokens when the promt is larger than 1024 tokens.
https://openai.com/index/api-prompt-caching/

Is this feature enable in Azure OpenAI? If it's not, is there an ETA?

YutongTie-MSFT 52,596 Reputation points

2024-10-02T20:34:44.3333333+00:00

Hello @Christian

Thanks for reaching out to us, this feature is not available in Azure right now, since this feature was just released by OpenAI yesterday, there is no ETA released from Azure OpenAI side.

I can check with internally to see is there any near future plan, will let you know if there is one.

Regards,

Yutong
Tien Hoang 25 Reputation points

2024-10-15T08:26:55.9266667+00:00

Hi YutongTie-MSFT,

Please tell me know is prompt caching available on Azure or not right now?

Thank you.
Dexter Awoyemi 15 Reputation points

2024-10-20T20:30:53.8566667+00:00

Hi @YutongTie-MSFT - are there any timelines you're able to share? Thanks
Korita, Denada (K-DDP/3) 15 Reputation points

2024-10-22T11:16:51.9133333+00:00

Is there any update on this? @YutongTie-MSFT
Milkowski, Robert 15 Reputation points

2024-10-24T09:29:29.24+00:00

I still (tried this morning) don't see cached_tokens field returned for gpt-4o-2024=08-06 when querying Azure (it is there when querying OpenAI direct).
Abdellah Sabir 0 Reputation points

2024-10-25T14:31:06.7233333+00:00

I just tried and the caching still does not working for gpt-4o-2024=08-06

Accepted answer

Abel Wenning 81 Reputation points

2024-10-22T18:20:14.3933333+00:00
Yes, as of yesterday, per Microsoft:
Currently only the following models support prompt caching with Azure OpenAI:

o1-preview-2024-09-12

o1-mini-2024-09-12

Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/prompt-caching

Cc: @YutongTie-MSFT , @ Christian-7033 @TienHoang-5634, @Dexter Awoyemi , @koritadenadakddp3-4938 (sorry, I don't know how to effectively tag the other 3 [or 8] people)
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Saravanan Palanivel 5 Reputation points

2024-10-24T08:43:30.15+00:00
As of 23rd October, prompt caching is extended for following models as well

gpt-4o-2024-05-13

gpt-4o-2024-08-06

gpt-4o-mini-2024-07-18
Please sign in to rate this answer.
Milkowski, Robert 15 Reputation points

2024-10-24T09:30:31.7266667+00:00

I still (tried this morning) don't see cached_tokens field returned for gpt-4o-2024=08-06 when querying Azure (it is there when querying OpenAI direct).

Saravanan Palanivel 5 Reputation points

2024-10-24T09:31:32.94+00:00

At this time, only o1-preview-2024-09-12 and o1-mini-2024-09-12 models support the cached_tokens API response parameter.

Vitcu, Marvin 5 Reputation points

2024-10-24T11:21:58.1+00:00

Milkowski, Robert

this is expected at the moment. The documentation mentions:

Official support for prompt caching was first added in API version 2024-10-01-preview. At this time, only o1-preview-2024-09-12 and o1-mini-2024-09-12 models support the cached_tokens API response parameter.

Milkowski, Robert 15 Reputation points

2024-10-24T11:38:38.0366667+00:00

So what do they mean by that support has been added to gpt-4o as well? It uses prompt caching but doesn't include stats in its replies?

Vitcu, Marvin 5 Reputation points

2024-10-24T11:49:40.33+00:00

Milkowski, Robert

yes. I interpreted it the same as you. Apparently prompt caching is turned on for gpt-4o, but there is no way of telling.

You don't get any information about how many tokes were cached.

Since the latency also is currently unaffected by prompt caching (based on my initial testing), it's almost impossible to verify if you use prompt caching or not.

As a current "workaround": You could use o1-mini for setting up everything so it works best with prompt caching. With o1-mini you can get information about the cached tokens. At the end replace the model with gpt-4o again.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Prompt caching in Azure OpenAI?

1 additional answer

Your answer