Timmer trigger RecursionError colorama/ansitowin32.py", line 29

Question

Timmer trigger RecursionError colorama/ansitowin32.py", line 29

Mark Sanders 20

I created a timer trigger function that runs every 5 minutes. Python 3.9. Using the cli, I check the number of namespaces in a kubernetes cluster. Based on results, the node pool min sizes are adjusted.

It seems to run fine for most of the day. Suddenly, the function starts failing with the error below. I have to restart the function to get things working again. Any suggestions would be helpful.

Exception: RecursionError: maximum recursion depth exceeded
Stack:   File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 484, in _handle__invocation_request
    sys.stdout.flush()
  File "/home/site/wwwroot/.python_packages/lib/site-packages/colorama/ansitowin32.py", line 29, in __getattr__
    return getattr(self.__wrapped, name)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/colorama/ansitowin32.py", line 29, in __getattr__
    return getattr(self.__wrapped, name)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/colorama/ansitowin32.py", line 29, in __getattr__
    return getattr(self.__wrapped, name)
  [Previous line repeated 490 more times]

MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2023-02-23T22:02:38.76+00:00

Mark Sanders Thank you for posting your question in Microsoft Q&A. Based on the stack trace above, it appears to be an issue related to https://github.com/tartley/colorama library and found there are multiple open issues/discussions for RecursionError in the repo.
PR#198 specifically talks about this exception being raised by __getattr__() and the workaround is to not wrap the same stream multiple times as suggested. Please review your code (and other thread) and avoid calling it recursively.

If you have any specific questions or issues related to GH repo, I suggest you open a new issue in the repo. Feel free to add a comment for any questions in Azure Functions. Would be happy to answer any. I hope this helps with your question.
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2023-02-27T15:33:53.4933333+00:00

Mark Sanders Just a follow up on the comment above and see if it helped with your question.
Mark Sanders 20 Reputation points

2023-06-15T11:45:49.52+00:00

@MuthuKumaranMurugaachari-MSFT Sorry I am just getting back to you. I never got a notification that someone responded. I guess I need to "follow" my own question.

I appreciate the information you provided. I still have not been able to resolve this based on the links you shared. I have tried a number of things. What makes this difficult is that in only happens about ever 14 hours - consistently. It does not happen every time the functions are triggered.

I'll keep digging into it, and will post back if I find a soltion.

Mark Sanders 20

@MuthuKumaranMurugaachari-MSFT

I have looked through my code. I'm not positive, but I think this invoke call my be the root cause. I can't find a way to issue the kubectl command without calling invoke again in get_namespaces_cmd

    def get_namespace_count(self, az_cli, resource_group, cluster) -> int:
        """ Gets all namespaces, counts them minus aks-command, and returns count  """
        namespace_current_count = 0
        get_namespaces_cmd = (
            'aks', 'command', 'invoke', '-g', resource_group, '-n', cluster, '-c',
            "kubectl get ns --no-headers -o custom-columns=':metadata.name'", '--output', 'json', '--query', 'logs')

        try:
            az_cli.invoke(get_namespaces_cmd) # This line may be root cause

Your answer

MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2023-02-23T22:02:38.76+00:00

Mark Sanders Thank you for posting your question in Microsoft Q&A. Based on the stack trace above, it appears to be an issue related to https://github.com/tartley/colorama library and found there are multiple open issues/discussions for RecursionError in the repo.
PR#198 specifically talks about this exception being raised by __getattr__() and the workaround is to not wrap the same stream multiple times as suggested. Please review your code (and other thread) and avoid calling it recursively.

If you have any specific questions or issues related to GH repo, I suggest you open a new issue in the repo. Feel free to add a comment for any questions in Azure Functions. Would be happy to answer any. I hope this helps with your question.
MuthuKumaranMurugaachari-MSFT 22,441 Reputation points Moderator

2023-02-27T15:33:53.4933333+00:00

Mark Sanders Just a follow up on the comment above and see if it helped with your question.
Mark Sanders 20 Reputation points

2023-06-15T11:45:49.52+00:00

@MuthuKumaranMurugaachari-MSFT Sorry I am just getting back to you. I never got a notification that someone responded. I guess I need to "follow" my own question.

I appreciate the information you provided. I still have not been able to resolve this based on the links you shared. I have tried a number of things. What makes this difficult is that in only happens about ever 14 hours - consistently. It does not happen every time the functions are triggered.

I'll keep digging into it, and will post back if I find a soltion.
Mark Sanders 20 Reputation points

2023-08-09T10:24:40.6566667+00:00

@MuthuKumaranMurugaachari-MSFT

I have looked through my code. I'm not positive, but I think this invoke call my be the root cause. I can't find a way to issue the kubectl command without calling invoke again in get_namespaces_cmd

def get_namespace_count(self, az_cli, resource_group, cluster) -> int: """ Gets all namespaces, counts them minus aks-command, and returns count """ namespace_current_count = 0 get_namespaces_cmd = ( 'aks', 'command', 'invoke', '-g', resource_group, '-n', cluster, '-c', "kubectl get ns --no-headers -o custom-columns=':metadata.name'", '--output', 'json', '--query', 'logs') try: az_cli.invoke(get_namespaces_cmd) # This line may be root cause

Share via

Timmer trigger RecursionError colorama/ansitowin32.py", line 29

Your answer