Logging of prompts for admin review on Azure OpenAI Service

Question

Hello everyone,

I have deployed internally an Azure OpenAI endpoint. I however need to know what my users sent to it for compliance reasons.

It does not seem to be the case that the product has a built-in feature to record those prompts and allow admins to read them afterwards.

I could do it myself by setting a proxy server, and users would first send prompts to it, logging prompts here, before sending the prompts to my internal Azure OpenAI endpoint, but it would be extra work.

Do you think this feature of prompt logging is coming any time soon?

Best,

Daniel

Answer 1

@Dh1 While a lot of data is made available for monitoring purposes, the actual payloads for requests/responses are not persisted by default.

You would need to implement your own service which can perform this by intercepting requests/responses as you've mentioned and storing them appropriately.

You can refer to the sample web app used for OpenAI on your data as reference, where conversations are persisted into CosmosDB.