My function in my function app executes, but doesn't execute, when running in a docker container in kubernetes

Question

I am having trouble with a container runtime in a k8s cluster. I'm stuck.

Context: My org is moving away from using azure function apps as our production API runtime and moving toward using containerized web applications in managed AKS clusters.

In order to prevent work stoppage, we settled on a middle ground where we run our codebase inside of docker containers running the azure functions runtime.

Here's my dockerfile (very vanilla ... please note that I'm using dotnet 6.0 and in-process model which I hope to get away from very very soon):

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS installer-env
COPY . /src
RUN dotnet restore "/src/AxonVR.Azure.Authentication/AxonVR.Azure.Authentication.csproj"
RUN dotnet build "/src/AxonVR.Azure.Authentication/AxonVR.Azure.Authentication.csproj" -c Release -o "/bin/build"
RUN dotnet publish "/src/AxonVR.Azure.Authentication/AxonVR.Azure.Authentication.csproj" -c Release -o "/home/site/wwwroot"
FROM mcr.microsoft.com/azure-functions/dotnet:4
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
    AZURE_FUNCTIONS_ENVIRONMENT=Production
COPY --from=installer-env ["/home/site/wwwroot", "/home/site/wwwroot"]

The most fundamental operation this application offers is an IDP integrated oauth login. When someone hits our function app URI in a browser, they're redirected to an IDP where our system can operate as the middle man and an oauth token issuer for the IDP it is proxying. As half that process, after authenticating against the IDP, the user is sent in the browser back to a URI that has another function app running behind it. The function app running under this URI processes the redirect from the IDP and directs the browser to place the token in browser local storage, or in a cookie, or wherever.

namespace controllers 
{ 
    public class UserAuthController
    {
        public const string RootUri = "api/v3/auth/user";
        private readonly ILogger<UserAuthController> _logger;

        public UserAuthController(ILogger<UserAuthController> log)
        {
            _logger = log;

        }
     
        [FunctionName("UserAuthRedirect")]
		public async Task<IActionResult> UserAuthRedirect(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = RootUri + "/redirect")] HttpRequest req)
        {
            _logger.LogWarning("UserAuthRedirect triggered.");
            
			// business logic here
        }
		
		// other functions here
	}
}

But none of this happens. The function app body is simply not executing. I have logging statements all over the code that aren't being reported out. The function app runtime is returning a 500 http response, but nothing is logged in application logs. I have reduced the logging level app-wide to Trace but still get nothing.

This is a sample of the logs that the application runtime is offering up. They're anything but helpful.

      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'
dbug: Microsoft.AspNetCore.Routing.RouteBase[1]
      Request successfully matched the route with name 'UserAuthRedirect' and template 'api/v3/auth/user/redirect'

Prior to deploying this in a k8s cluster, I obviously tested it locally. The most suitable docker runtime I have on my local machine is docker compose. That works.

When I try this flow in docker compose, the request is processed normally, same as any other, and i get loads of logs. I get nothing when deployed in k8s. I can also just hit this URI directly and get a response with loads of logs without going through the full login flow

I don't understand why the azure function runtime in this docker container is failing to execute, and even more, just failing to log anything. There's no indication anywhere in this runtime as to what could be causing failure. Executing locally is different than running in the vr ag1 cluster. But those logs above are all I have, and they're totally useless.

Like I said, I have tried reducing the log-level to trace, I have double check that console logging is enabled in my host.json config.

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "default": "Trace",
      "Host.Aggregator": "Trace",
      "Host.Results": "Trace",
      "Function": "Trace",
      "Microsoft": "Trace",
      "Worker": "Trace",
      "Function.UserAuthRedirect.User": "Trace"
    }
  },
  "extensions": {
    "http": {
      "routePrefix": ""
    }
  }
}

I really don't know what to do at this point. It just makes no sense because it's the same docker image and same functions runtime that runs when i start the container on my local machine. It's driving me crazy. If anyone has any ideas, I would love to hear them. Thanks in advance. :-)

Answer 1

Thanks to Landon Hemsley, the root cause of the issue lied in Immutable Kubernetes Secret. In Landon's initial configuration, he had enabled the following setting

AzureWebJobsSecretStorageType=kubernetes

Created a kubernetes secret as below-

apiVersion: v1

kind: Secret

metadata:

  name: vrbe-auth-fn-keys

  namespace: vrbe

type: Opaque

immutable: true   # This is what caused the issue

data:

  host.master: <base64-key>

  functions.userauthredirect.default: <base64-key>

However, in this case the Azure Functions runtime attempted to write to the Kubernetes secret when initializing function keys even if the keys already exist.

Why?- When the secret is marked immutable, the runtime fails silently resulting in a 500 Internal Server Error and no function execution.

what was done to fix this issue?

Updated the secret as below-

Once the user removed the immutable key out of the secret, the user was able to make the request and get a response.

apiVersion: v1

kind: Secret

metadata:

  name: vrbe-auth-fn-keys

  namespace: vrbe

type: Opaque

data:

  host.master: <base64-key>

  functions.userauthredirect.default: <base64-key>

It’s worth noting that there is very little official documentation about how to properly structure Kubernetes secrets for Azure Functions when using AzureWebJobsSecretStorageType=kubernetes. A request has been taken to update the documentation for the same.

Thanks to everyone who took a look. Hopefully this helps the community to avoid the same pitfall.