How do you configure logging properly in a multi threaded python function app?

Question

How do you configure logging properly in a multi threaded python function app?

Nick Petzold 5

I'm building a python function app which utilises multithreading for performance reasons and have run into issues getting the logs to be emitted properly from non-main threads.

I've found this Stack Overflow Answer, which solves the issue, however when I add the azure-functions-worker to my requirements.txt, I get the following warning:

# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues

azure-functions-worker==1.1.9
...

I'm not detecting any unusual behaviour as a result, but obviously wary about doing something the platform specifically warns against. It's also imperative that I have comprehensive logging from the non-main threads, so I'd be very keen to find a proper solution to this.

Thanks in advance!

Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2023-05-10T16:26:50.0366667+00:00

Hi @Nick Petzold just circling back to see if the provided post helped resolve your issue. Let us know if you have any further questions on this topic so we can better assist you.
Nick Petzold 5 Reputation points

2023-05-11T12:04:26.85+00:00

Hi Ryan, thanks for your suggestion - I haven't had an opportunity to try it out just yet, but will give you a shout if we have any issues.

1 answer

Your answer

Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2023-05-10T16:26:50.0366667+00:00

Hi @Nick Petzold just circling back to see if the provided post helped resolve your issue. Let us know if you have any further questions on this topic so we can better assist you.
Nick Petzold 5 Reputation points

2023-05-11T12:04:26.85+00:00

Hi Ryan, thanks for your suggestion - I haven't had an opportunity to try it out just yet, but will give you a shout if we have any issues.

Answer 1

Ryan Hill 30,281 Microsoft Employee Moderator

The reason you're receiving the warning is because it's automatically managed by the platform. It gets updated with new features and bug fixes. You can read more at Azure Functions Python worker dependencies. Using an older/different version could have unintended consequences.

If logging via

import logging

logging.basicConfig(level=logging.DEBUG)

isn't working as expected, you can also look at monitoring through Application Insights.

import logging
from azure.functions.logging import AppInsightsHandler

app_insights_handler = AppInsightsHandler()
logging.getLogger().addHandler(app_insights_handler)

And leverage the logging object for .info(), .warning(), and .error().

Nick Petzold 5 Reputation points

2023-05-24T10:18:49.5166667+00:00

Hi @Ryan Hill, I've tried using AppInsightsHandler, however I'm unable to import the class. It also doesn't appear in any documentation online. Is it definitely a real thing?
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2023-06-07T14:31:13.7566667+00:00

You're right @Nick Petzold , I checked the SDK and did not find any references. My apologies for the erroneous code. I'm testing this on my own environment and will let you know my findings.
Maria Lombardi 0 Reputation points

2023-06-08T14:33:16.4566667+00:00
Hi all. Same issue also for me. The logging from the threads does not work.

The documentation says this (copy paste from the docs):

import azure.functions as func import logging import threading def main(req, context): logging.info('Python HTTP trigger function processed a request.') t = threading.Thread(target=log_function, args=(context,)) t.start() def log_function(context): context.thread_local_storage.invocation_id = context.invocation_id logging.info('Logging from thread.')

In my case, I receive the following error:

[2023-06-08T13:48:51.436Z] AttributeError: 'Context' object has no attribute 'thread_local_storage'

Really not understood why. If you have any hint, I'll really appreciate!

Many thanks.

Maria
Nick Petzold 5 Reputation points

2023-06-22T13:32:52.16+00:00

Hi @Ryan Hill, any update on this?
Matheus Nishida 0 Reputation points

2023-09-14T19:53:13.05+00:00

Had the same issue here, and also tried the same doc as Maria but got the same error. Any ideas here?
Nathan George 0 Reputation points

2024-03-14T00:53:52.24+00:00

I'm not finding AppInsightsHandler anywhere in the codebase, and it seems azure.functions.logging no longer exists. I did find this stackoverflow on it, but can't install grpcio on Linux. https://stackoverflow.com/questions/55934085/how-to-redirect-logs-from-secondary-threads-in-azure-functions-using-python

Nathan George 0

Ok, this seems to work: https://github.com/Azure/azure-functions-python-worker/issues/1358#issuecomment-1858604425

Briefly:

import azure.functions as func
import threading
import logging

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)

@app.route(route="http_trigger")
def http_trigger(req: func.HttpRequest, context: func.Context) -> func.HttpResponse:
    t1 = threading.Thread(target=thread_function, args=(context, 'Thread1 used.'))
    t1.start()
    
    return func.HttpResponse("Ok", status_code=200)
    
def thread_function(context: func.Context, message: str):
    context.thread_local_storage.invocation_id = context.invocation_id
    for _ in range(10):
        logging.info(message)

Share via

How do you configure logging properly in a multi threaded python function app?

1 answer

Your answer