Clarification on Prompt Cache: Difference Between in_memory and 24h Retention

Question

Hello Microsoft Team,

I have a question regarding the Prompt Cache feature in Azure OpenAI / Azure AI Foundry.

Based on the documentation, I understand that there are two prompt cache retention modes:

• Default in_memory
• prompt_cache_retention: "24h"

My understanding is that the primary difference is the retention duration. However, I would like to confirm whether there are any architectural or behavioral differences beyond cache lifetime.

Specifically:

1. Is the only difference between in_memory and 24h the cache retention time?
2. Does the 24h retention mode survive situations where the original GPU memory is evicted, the serving instance is restarted, or requests are routed to another serving instance?
3. Is the 24h cache stored in a persistent cache layer managed by the platform, or is it still purely an in-memory cache with an extended lifetime?
4. From a cache hit perspective, can 24h achieve a higher hit rate because it is independent of GPU memory eviction?

For example, if I process thousands of requests over several hours using the same system prompt and prefix, should I expect the 24h retention mode to continue producing cache hits even if the backend serving infrastructure changes?

I would appreciate clarification on whether the difference is only the retention time, or whether the underlying cache implementation is also different.

Thank you!Hello Microsoft Team,

I have a question regarding the Prompt Cache feature in Azure OpenAI / Azure AI Foundry.

Based on the documentation, I understand that there are two prompt cache retention modes:

• Default in_memory
• prompt_cache_retention: "24h"

My understanding is that the primary difference is the retention duration. However, I would like to confirm whether there are any architectural or behavioral differences beyond cache lifetime.

Specifically:

1. Is the only difference between in_memory and 24h the cache retention time?
2. Does the 24h retention mode survive situations where the original GPU memory is evicted, the serving instance is restarted, or requests are routed to another serving instance?
3. Is the 24h cache stored in a persistent cache layer managed by the platform, or is it still purely an in-memory cache with an extended lifetime?
4. From a cache hit perspective, can 24h achieve a higher hit rate because it is independent of GPU memory eviction?

For example, if I process thousands of requests over several hours using the same system prompt and prefix, should I expect the 24h retention mode to continue producing cache hits even if the backend serving infrastructure changes?

I would appreciate clarification on whether the difference is only the retention time, or whether the underlying cache implementation is also different.

Thank you!