AAD B2C Example Only Fails to Authenticate in Kubernetes/Ingress

asked 2020-09-29T23:31:34.453+00:00
Siegfried Heintze 1,306 Reputation points

I have run the Microsoft AAD B2C Sample webapp (called todolistclient) from GitHub (see 4-2-B2C) with some small enhancements such as using a redis server to cache AAD B2C authentication tokens. (As discussed in how-to-use-aad-with-kubernetes.html, I'm using the caching approach instead of OAUTH2_PROXY). This is working on my desktop development machine and is successfully authenticating me (using my AAD B2C tenant) when using my Microsoft Email account and my facebook account (with no docker/kubernetes).

I have been also following Create an ingress controller with a static public IP address in Azure Kubernetes Service (AKS) to deploy this to a kubernetes cluster.

Now when I launch chrome from cygwin bash using "chrome https://$PUBLIC_TODO_AKS_DNS/todolistclient" where todolistclient is the path in ingress and PUBLIC_TODO_AKS_DNS is the environment variable that contains the AKS domain name, my AADB2C tenant briefly appears in the browser's address bar indicating that it is successfully passing thru the kubernetes/ingress proxy and my todolistclient is trying to authenticate with my AADB2C tenant (indicating that I have updated my the reply URL in my tenant correctly by inserting "/todolistclient" immediately prior to "/signin-oidc").

However I get this in the browser:

Error.  
An error occurred while processing your request.  
Request ID: |a24109a5-489efbc4165cb4ae.  
  
Development Mode  
Swapping to Development environment will display more detailed information about the error that occurred.  
  
Development environment should not be enabled in deployed applications, as it can result in sensitive information from exceptions being displayed to end users. For local debugging, development environment can be enabled by setting the ASPNETCORE_ENVIRONMENT environment variable to Development, and restarting the application.  
  
© 2018 - WebApp_OpenIDConnect_DotNet  
  

OK, I go back to the source and modify the configure function to always be in dev mode:

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IHostApplicationLifetime lifetime, IDistributedCache cache)  
        {  
            app.UsePathBase(new Microsoft.AspNetCore.Http.PathString("/todolistclient"));  
            if (true || env.IsDevelopment())  
            {  
                app.UseDeveloperExceptionPage();  
            }  
            else  
            {  
                app.UseExceptionHandler("/Home/Error");  
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.  
                app.UseHsts();  
            }  

I explicitly use VS2019 to rebuild and publish and I get the same error message from the browser...

How could this be? It looks like my changes to always use dev mode were never deployed. However, I check Dockerhub and see that I have a recent deployment and the digest matches the digest I see when I describe the todolistclient pod.
VS2019 said the rebuild was successful and the deployment was successful...

Whoops! I see the instructions concerning ASPNETCORE_ENVIRONMENT so I edit my yaml for the Kubernetes deployment and service and redeploy again... No luck... Same error message.

Also have deployed another another ASP.NET Core Sample WebApp callled KubernetesHelloASPNET that does not authenticate to the same Ingress/Kubernetes cluster (with a different ingress path, of course). This works fine.

So I deleted my AKS cluster to save money. If the Microsoft AAD team would like me to bring it up again so you can reproduce the problem, let me know. I was hesitating to publish my domain name to the public internet for security reasons.

Thanks

Siegfried

Tue Sep 29 2020 Evening Update

I forgot to use kubectl to look at the pod logs... Looks like I'm not authenticating.... It does not prompt me for username and password possibly because it is remembering my creds via some cookies. I still don't know what is wrong, however. This todolistclient authenticates via AAD B2C on my desktop with no docker/kubernetes just fine.

^[[40m^[[1m^[[33mwarn^[[39m^[[22m^[[49m: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]  
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[0]  
      User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]  
      Creating key {79d67135-9201-4407-bb1d-9ff9177226fc} with creation date 2020-09-29 22:48:37Z, activation date 2020-09-29 22:48:37Z, and expiration date 2020-12-28 22:48:37Z.  
^[[40m^[[1m^[[33mwarn^[[39m^[[22m^[[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]  
      No XML encryptor configured. Key {79d67135-9201-4407-bb1d-9ff9177226fc} may be persisted to storage in unencrypted form.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[39]  
      Writing data to file '/root/.aspnet/DataProtection-Keys/key-79d67135-9201-4407-bb1d-9ff9177226fc.xml'.  
^[[40m^[[1m^[[33mwarn^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Server.Kestrel[0]  
      Overriding address(es) 'http://+:80'. Binding to endpoints defined in UseKestrel() instead.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.Hosting.Lifetime[0]  
      Now listening on: http://0.0.0.0:80  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.Hosting.Lifetime[0]  
      Now listening on: http://0.0.0.0:443  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.Hosting.Lifetime[0]  
      Application started. Press Ctrl+C to shut down.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.Hosting.Lifetime[0]  
      Hosting environment: Development  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.Hosting.Lifetime[0]  
      Content root path: /app  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
      Request starting HTTP/1.1 GET http://siegdnsname.westus2.cloudapp.azure.com/todolistclient/MicrosoftIdentity/Account/Error    
^[[40m^[[1m^[[33mwarn^[[39m^[[22m^[[49m: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]  
      Failed to determine the https port for redirect.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[7]  
      OpenIdConnect was not authenticated. Failure message: Not authenticated  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]  
      Executing endpoint '/Account/Error'  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[3]  
      Route matched with {page = "/Account/Error", area = "MicrosoftIdentity", action = "", controller = ""}. Executing page /Account/Error  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[101]  
      Executing handler method Microsoft.Identity.Web.UI.Areas.MicrosoftIdentity.Pages.Account.ErrorModel.OnGet - ModelState is Valid  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[102]  
      Executed handler method OnGet, returned result .  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[103]  
      Executing an implicit handler method - ModelState is Valid  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[104]  
      Executed an implicit handler method, returned result Microsoft.AspNetCore.Mvc.RazorPages.PageResult.  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker[4]  
      Executed page /Account/Error in 105.3644ms  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]  
      Executed endpoint '/Account/Error'  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
      Request finished in 652.2536ms 200 text/html; charset=utf-8  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
      Request starting HTTP/1.1 GET http://siegdnsname.westus2.cloudapp.azure.com/todolistclient/css/site.css    
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
      Request starting HTTP/1.1 GET http://siegdnsname.westus2.cloudapp.azure.com/todolistclient/lib/bootstrap/dist/js/bootstrap.js    
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]  
      Sending file. Request path: '/lib/bootstrap/dist/js/bootstrap.js'. Physical path: '/app/wwwroot/lib/bootstrap/dist/js/bootstrap.js'  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]  
      Sending file. Request path: '/css/site.css'. Physical path: '/app/wwwroot/css/site.css'  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
      Request finished in 9.0463ms 200 application/javascript  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
      Request finished in 11.4568ms 200 text/css  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
      Request starting HTTP/1.1 GET http://siegdnsname.westus2.cloudapp.azure.com/todolistclient/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.woff2    
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]  
      Sending file. Request path: '/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.woff2'. Physical path: '/app/wwwroot/lib/bootstrap/dist/fonts/glyphicons-halflings-regular.woff2'  
^[[40m^[[32minfo^[[39m^[[22m^[[49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
      Request finished in 3.0211ms 200 font/woff2  
  

Mon Oct 05 2020 Morning Update

I've been trying to cut and paste this code from KrishnenduGhosh's link and I'm having some trouble.

var storageAccount = CloudStorageAccount.Parse("<storage account connection string">);  
var client = storageAccount.CreateCloudBlobClient();  
var container = client.GetContainerReference("<key store container name>");  
  
var azureServiceTokenProvider = new AzureServiceTokenProvider();  
var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(  
        azureServiceTokenProvider.KeyVaultTokenCallback));  
  
services.AddDataProtection()  
    //This blob must already exist before the application is run  
    .PersistKeysToAzureBlobStorage(container, "<key store blob name>")  
    //Removing this line below for an initial run will ensure the file is created correctly  
    .ProtectKeysWithAzureKeyVault(keyVaultClient, "<keyIdentifier>");  

(1) PersistKeysToAzureBlobStorage does not compile and I'm wondering which override I should use. I was thinking about calling it with the

(a) blob connection string,

(b) a container name string (is the same name I passed to GetContainerReference and used with the "az storage container create command"?) and

(c) the blob name (is this just a name I make up that is never used again?).

(2) ProtectKeysWithAzureKeyVault does not compile either. It cannot convert the keyVaultClient to a system.Uri, and cannot convert my keyVaultKeyname string to a Azure.Core.TokenCredential. Can you provide me some guidance on providing a URI and creating a TokenCredential? There are many descendants of TokenCredential. Which one would be appropriate when running inside the ConfigureServices functions of an ASP.NET Core Web App inside a K8 replica set?

Tue Nov 10 2020 Morning Update:

This code in the updated documentation seems to be working in the sense the pod logs indicate that the ASP.NET core is not generating any error messages in the Startup function any more and I see it created a blob called keys.xml:

services.AddDataProtection()  
    //This blob must already exist before the application is run  
    .PersistKeysToAzureBlobStorage("<storage account connection string", "<key store container name>", "<key store blob name>")  
    //Removing this line below for an initial run will ensure the file is created correctly  
    .ProtectKeysWithAzureKeyVault(new Uri("<keyIdentifier>"), new DefaultAzureCredential());  

Here is my actual code as printed in my pod log:

var dpd= services.AddDataProtection();  
dpd=dpd.PersistKeysToAzureBlobStorage(connectionString: storageConnection, containerName: "dataprotection", blobName: "keys.xml");  
//dpd=dpd.ProtectKeysWithAzureKeyVault(keyIdentifier: new Uri("https://keyvaultname.vault.azure.net/keys/keyvaultkeyname/") /*vault key identifier used for key encryption*/, tokenCredential: new DefaultAzureCredential()) ;  
services.AddDistributedMemoryCache();  

As per the comments I commented out the call to ProtectKeysWithAzureKeyVault. However, when I visit the web site I see these errors in the pod log:

 Request starting HTTP/1.1 GET http://xxxxxxxxxxx.westus2.cloudapp.azure.com/todolistclient    
 [40m [1m [33mwarn [39m [22m [49m: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]  
    Failed to determine the https port for redirect.  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[7]  
      OpenIdConnect was not authenticated. Failure message: Not authenticated  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService[2]  
      Authorization failed.  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[12]  
      AuthenticationScheme: OpenIdConnect was challenged.  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
      Request finished in 1375.6286ms 302   
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
      Request starting HTTP/1.1 GET http://xxxxxx.westus2.cloudapp.azure.com/todolistclient/MicrosoftIdentity/Account/Error    
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[7]  
      OpenIdConnect was not authenticated. Failure message: Not authenticated  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]  
      Executing endpoint '/Account/Error'  

Is this an AAD problem or an azure storage problem or ASP.NET DataProtection problem? I thought about using fiddler to look at the traffic, but I don't know how to do that when it is running inside a K8 cluster...

The browser display indicated that I should turn on developer mode by setting ASPNETCORE_ENVIRONMENT=Development and I confirmed that this is true with my log statements in Startup. Nevertheless, the browser display did not provide any helpful details and indicated that I was not in developer mode.

Here is the actual error on the browser display:

An error occurred while processing your request.  
Request ID: |4323a307-4096636ba1555539.  

Development Mode  
Swapping to Development environment will display more detailed information about the error that occurred.  

Development environment should not be enabled in deployed applications, as it can result in sensitive information from exceptions being displayed to end users. For   
local debugging, development environment can be enabled by setting the ASPNETCORE_ENVIRONMENT environment variable to Development, and restarting the   
application.  

© 2018 - WebApp_OpenIDConnect_DotNet  

How do I turn developer mode on?

Mon Dec 14 2020 Update:

Progress: I am using redis to cache my AADB2C tokens and apparently Kubernetes requires my redis server to be in the same namespace as my todolistclient and the stack trace for the failure to connect to the redis server does not show up in the kubernetes pod logs for some strange reason... But it does show up in Bridge to Kubernetes... That problem is now fixed...

So I configure and deploy my web app and configure ingress and point my EDGE browser at my todolistclient via the ingress controller. I get the unhelpful display above
in the browser because developer mode is turned off and apparently it cannot be turned on while in Kubernetes. I check my browser for the cookies for the site and I delete the single cookie and try again... No change... At no time does my AADB2C tenant prompt me for credentials like it used to do when running with no docker and no kubernetes.

I believe this is a AADB2C problem. I have a public static IP and domain for my exposed ingress controller inside my kubernetes cluster. The ingress controller seems to be configured correctly because it is successfully forwarding traffic to the todolistclient and some other test sites inside my AKS cluster.

From the todolistclient pod logs I see:  
      Request starting HTTP/1.1 POST http://xxxxxxxxxxxxxx.westus2.cloudapp.azure.com/todolistclient/signin-oidc application/x-www-form-urlencoded 754  
 [40m [1m [33mwarn [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[15]  
      '.AspNetCore.Correlation.OpenIdConnect.qbq-89SbFqA57KLy9tkNPeqsUmuukOrdAoCLv_CAA4Q' cookie not found.  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[4]  
     Error from RemoteAuthentication: Correlation failed..  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Hosting.Diagnostics[2]  
   Request finished in 22.7751ms 302   
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Hosting.Diagnostics[1]  
    Request starting HTTP/1.1 GET http://xxxxxxxxxxxx.westus2.cloudapp.azure.com/todolistclient/MicrosoftIdentity/Account/Error    
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectHandler[7]  
   OpenIdConnect was not authenticated. Failure message: Not authenticated  
 [40m [32minfo [39m [22m [49m: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]  
  Executing endpoint '/Account/Error'  

In the ingress pod logs I see this:

10.244.1.1 - - [15/Dec/2020:02:18:57 +0000] "GET /todolistclient/MicrosoftIdentity/Account/SignIn HTTP/2.0" 302 0   
"https://xxxxxxxxxxx.westus2.cloudapp.azure.com/todolistclient/MicrosoftIdentity/Account/Error" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36   
(KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.60" 57 0.006 [ingress-basic-todolistclient-80] [] 10.244.0.7:80 0 0.004 302       
<bighexnumber>  
10.244.1.1 - - [15/Dec/2020:02:18:57 +0000] "POST /todolistclient/signin-oidc HTTP/2.0" 302 0 "https://mytenantname.b2clogin.com/" "Mozilla/5.0 (Windows NT   
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.60" 792 0.002 [ingress-basic-todolistclient-80] []   
10.244.0.7:80 0 0.000 302 8 <another big hex number>  
10.244.1.1 - - [15/Dec/2020:02:18:57 +0000] "GET /todolistclient/MicrosoftIdentity/Account/Error HTTP/2.0" 200 1044 "https://myaadb2ctenant.b2clogin.com/"   
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.60" 56 0.006 [ingress-basic-   
todolistclient-80] [] 10.244.0.7:80 2788 0.008 200 <yetanotherbighexnumber>  

The AADB2C callback looks correct, the client web site URLs look, and my tenant URL looks correct...
So it is not authenticating because "Correlation failed..." what does that mean and how do I fix it?

Fri Jan 08 2021 Update:

Progress: I can now run the example 4-2-B2C inside Azure Kubernetes and authenticate the client...

(1) How do I confirm that my calls to AddDataProtection...PersistKeysToAzureBlobStorage...PersistKeysToAzureBlobStorage are working? I enhanced the C# client to add a cookie and it looks like the cookie is persisting in between sesssions... I was hoping to see evidence of this by downloading the blob but it was not obvious after looking at the contents of the blob... Is there a better way to confirm that DataProtection is working?

(2) Now I want to run the REST server part as a kubernetes service... Is the technique the same?
(2A) Add a calls to AddDataProtection...PersistKeysToAzureBlobStorage...PersistKeysToAzureBlobStorage and specify a different blob name?
(2B) Do I need to also call AddDistributedRedisCache like I do in the client? Can I use the same redis cache as I use for the client?

Thank you

Siegfried

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,103 questions
Azure Active Directory External Identities
{count} votes