Transfer FunctionAppLogs to Datadog, how to solve the JSON log content has been escaped.

Question

We deployed a Function App by code to App Service Plan. The Function is nodejs, and through winston logger to output a JSON to console, and we setup log transfer to Datadog in the diagnostic setting through event hub trigger Datadog Function to transfer log to Datadog site.

In our appliation log stream, we can see the logs output as below:

{"dd":{"env":"test","service":"my-service","span_id":"6172404353039023300","trace_id":"6172404353039023300","version":"1.0.0"},"level":"info","message":"Http function processed request for url \"https://dxherohttpexample.azurewebsites.net/api/httpTrigger\"","requestUrl":"https://dxherohttpexample.azurewebsites.net/api/httpTrigger","timestamp":"2024-08-21T16:01:07.350Z"}

When we check the log stream in the Datadog Function App before transfer, the logs were be added some Azure information and escaped , we can see in the properties.message the result shows \' instead of ":

{"properties":"{'appName':'dxherohttpexample','roleInstance':'03aaff03b57819dc178782387057ed48ba739dd89e2a9b1dcc22e36b6a0b4ace','message':'{\\'dd\\':{\\'env\\':\\'test\\',\\'service\\':\\'my-service\\',\\'span_id\\':\\'6172404353039023300\\',\\'trace_id\\':\\'6172404353039023300\\',\\'version\\':\\'1.0.0\\'},\\'level\\':\\'info\\',\\'message\\':\\'Http function processed request for url \\\\\\'https://dxherohttpexample.azurewebsites.net/api/httpTrigger\\\\\\'\\',\\'requestUrl\\':\\'https://dxherohttpexample.azurewebsites.net/api/httpTrigger\\',\\'timestamp\\':\\'2024-08-21T16:01:07.350Z\\'}','category':'Host.Function.Console','hostVersion':'4.35.1.1','hostInstanceId':'7ec64f14-ee6a-40a6-ad6d-b40734fd4e52','level':'Information','levelId':2,'processId':27}","time":"08/21/2024 16:01:07",
...
...
}

These logs cannot be recognized by the JSON processor, so the log content cannot be parsed by the Datadog pipeline. Is there any way I can adjust the settings on the Azure Function App side to prevent escaping?

Answer 1

Hello Luke Yen,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having challenges transferring FunctionAppLogs to Datadog, and you need how to solve the JSON log content that has been escaped.

1. To prevent the escaping of logs in your Azure Function App, you can adjust the settings in the host.json file by modify it and configure it to handle logs properly without escaping characters. For an example:

      {
        "version": "2.0",
        "logging": {
          "applicationInsights": {
            "samplingSettings": {
              "isEnabled": false
            }
          },
          "logLevel": {
            "Function": "Information",
            "default": "Information"
          }
        }
      }
   

2. Also, instead of relying on the default logging mechanism, you can implement a custom logger that formats the logs correctly before they are sent to Datadog.
3. About Datadog, direct integration with Azure Functions can help in bypassing the additional Azure metadata that gets added to the logs. https://learn.microsoft.com/en-us/azure/azure-functions/configure-monitoring https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring
4. Also, adjust the serialization settings in the Event Hub configuration to maintain the original log structure.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam