How can I log prompt inputs to Azure OpenAI to enable full-prompt logging in Log Analytics?

Sikora, Nicholas 5

My goal is use Azure Monitor to log and monitor full text prompt inputs to the Azure OpenAI service. Is this possible and if so, how can we configure / enable the ability log prompt inputs to the Azure OpenAI service?

I already have a Log Analytics Workspace but for compliance reasons have to ensure that specific content and keywords are not included in prompts.

romungi-MSFT 42,206 Reputation points Microsoft Employee

2023-03-27T07:22:31.46+00:00

Sikora, Nicholas I believe you need to setup diagnostic logging of your resource to get resource level logs. I am not quite sure if this enables collection of prompt inputs though. I will try to check internally and confirm the same. Thanks!!
Clive Watson 5,716 Reputation points MVP

2023-03-29T17:25:42.3566667+00:00
In my (limited) testing the AzureDiagnostic and AzureActivity logs enabled by diagnostic logging dont contain the words used from the Playground (the text you enter that generates the tokens).

Simple KQL example:

AzureDiagnostics | where TimeGenerated >= ago(1h) | where ResourceProvider =~ "MICROSOFT.COGNITIVESERVICES" | join kind=inner( AzureActivity | where TimeGenerated >= ago(1h) | where ResourceProviderValue =~ "MICROSOFT.COGNITIVESERVICES" ) on $left.ResourceProvider == $right.ResourceProviderValue
Fergus Kidd 0 Reputation points

2023-04-13T09:21:19.22+00:00

https://github.com/Azure-Samples/openai-python-enterprise-logging
Jim Richardson 5 Reputation points

2023-12-19T23:01:49.1866667+00:00

The Enterprise Logging above is technically correct in that it logs the prompts and completions...However, it doesn't seem to work with the OpenAI Studio Chat Playground. The Chat Playground seems to bypass the API Management service and so the prompts and completions aren't logged :(
Ben Gimblett 3,410 Reputation points Microsoft Employee

2023-12-27T11:01:11.6433333+00:00

The chat playground is as the name suggests a "playground" - it's not meant for a production app :)

The enterprise architecture pattern which is cited above utilises APIM as a middle tier component to funnel the client to Open AI requests...

The core feature presented by the article is the fact that you can "opt in" to having APIMs Az monitor configuration log "x" bytes of request/response body content to the Gateway access logs. Note: You can also ask App insights to do this too if you're using it in APIM

Some considerations here: The number of bytes to log, ensuring you DONT collect sensitive or PII data that will otherwise automatically get scraped into Log Analytics. It's generally recommended that log data should either have sensitive data removed or masked before persisting to log store.

In general you've 3 ways to utilise open ai
*In code
*Through an orchestrator like semantic kernel or langchain
*Through a workflow - for example promtpflow

In each case there's an opportunity to add components to log sanitized request/response content

In summary

APIM provides a neat out of the box solution, but it would require extra work (and possibly cross process calls to another service) to sanitize request/response content - if this is a risk/requirement (it wont be in all customer scenarios)