Windows container pods can't download signingkey

Question

hello,
We have several k8s clusters, about three months ago we started to have a problem with the signing in the windows pods in the Australian region, and two weeks ago started the same problem in the Europe region. The problem in the Europe region started without changes in the container. In the Europe region containers that have been running for several months began to have the problem.

We have an IIS which we have a sign using an azure AAD registered application, that uses this part of the code. Our IIS App runs on a Windows container Pod.

                            IdentityModelEventSource.ShowPII = true;  
  
                            LoggingTools.Logging.Log("1-Jwt, init SigningKeys");  
                            var _tenant = Startup.getCfgKey("Tenant");  
                            var _authority = string.Format(Startup.getCfgKey("Authority"), _tenant);  
  
                            foreach (string metadataAddress in new string[]  
                            {  
                            $"{_authority}/v2.0/.well-known/openid-configuration",  
                                // $"{_authority}/.well-known/openid-configuration"  
                            })  
                            {  
                                var retriever = new HttpDocumentRetriever  
                                {  
                                    RequireHttps = true,  
                                };  
  
                                var configurationRetriever = new OpenIdConnectConfigurationRetriever();  
  
                                try  
                                {  
                                    var configManager = new ConfigurationManager<OpenIdConnectConfiguration>(metadataAddress, configurationRetriever, retriever);  
  
                                    var config = configManager.GetConfigurationAsync().Result;  
                                    SigningKeys = config.SigningKeys;  
                                    ValidIssuer = config.Issuer;  
  
                                    LoggingTools.Logging.Log($"2-Jwt, SigningKeys, {SigningKeys.Count}, {metadataAddress}");  
                                    break;  
                                }  
                                catch (Exception ex)  
                                {  
                                    LoggingTools.Logging.Log($"error Jwt: {ex.Message}\n{ex.StackTrace}\n{metadataAddress}");  
                                }  
                            }  
  

The application cannot download the SigningKeys for a period of time that goes from 5 minutes to 15 minutes sometimes, according to the logs.

18:31:34 1-Jwt, init SigningKeys 18:31:34 error Jwt: One or more errors occurred. at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at VFCloudWebFront.JwtAuthModule.Init(HttpApplication context) 18:31:34 1-Jwt, init SigningKeys   
. . .  
18:37:30 1-Jwt, init SigningKeys  
  
18:37:30 2-Jwt, SigningKeys, 14  

 

Although we have a specific pod that can never download the signing keys. and we have noticed that in that specific pods if an nslookup is done to login.microsoftonline.com it fails

    C:\app\vf>nslookup login.microsoftonline.com  
    DNS request timed out.  
        timeout was 2 seconds.  
    Server:  UnKnown  
    Address:  12.0.0.10  
      
    DNS request timed out.  
        timeout was 2 seconds.  
    DNS request timed out.  
        timeout was 2 seconds.  
    DNS request timed out.  
        timeout was 2 seconds.  
    DNS request timed out.  
        timeout was 2 seconds.  
    *** Request to UnKnown timed-out   

But in those cases where it unlocks after 7 to 15 minutes, this does not happen, only the IIS apparently fails because if I run an application outside the IIS it does down the SigningKeys well.

C:\app\vf>nslookup login.microsoftonline.com  
Server:  kube-dns.kube-system.svc.cluster.local  
Address:  10.0.0.10  
  
Non-authoritative answer:  
Name:    www.tm.ak.prd.aadg.trafficmanager.net  
Addresses:  20.190.142.166  
          40.126.14.163  
          20.190.142.165  
          40.126.14.160  
          20.190.142.167  
          20.190.142.163  
          20.190.142.164  
          40.126.14.161  
Aliases:  login.microsoftonline.com  
          ak.privatelink.msidentity.com  
  

 

Answer 1

I did another test and it is trying to download the URL using WebClient c#. this fails, however, if I do it under the same URL outside of IIS it goes down fine. however, it still cannot be downloaded within IIS. This is a Windows Container running on Azure and has been running for over 1 year now with no problems.

C:\app\vf>type c:\Windows\Temp\vf_iis-2022-11-04.log
19:43:49 1-Python.Runtime.Runtime.PythonDLL: python39
19:43:50 2-init MapSignalR
19:43:50 3-Provider AZURE
19:43:52 1-Jwt, init SigningKeys
. . .

The remote server returned an error: (400) Bad Request.
19:44:12 1-Jwt, init SigningKeys
19:44:13 error Jwt: One or more errors occurred.
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at VFCloudWebFront.JwtAuthModule.DownSignKeys()
https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
19:44:13 fail check download https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
The remote server returned an error: (400) Bad Request.

C:\app\vf>time
The current time is: 19:45:08.92
Enter the new time:

C:\app\vf>curl https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
{"token_endpoint":"https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/oauth2/v2.0/token","token_endpoint_auth_methods_supported":["client_secret_post","private_key_jwt","client_secret_basic"],"jwks_uri":"https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/discovery/v2.0/keys","response_modes_supported":["query","fragment","form_post"],"subject_types_supported":["pairwise"],"id_token_signing_alg_values_supported":["RS256"],"response_types_supported":["code","id_token","code id_token","id_token
. . .

C:\app\vf>type c:\Windows\Temp\vf_iis-2022-11-04.log
19:43:49 1-Python.Runtime.Runtime.PythonDLL: python39
19:43:50 2-init MapSignalR
19:43:50 3-Provider AZURE
19:43:52 1-Jwt, init SigningKeys
. . .

The remote server returned an error: (400) Bad Request.
19:44:12 1-Jwt, init SigningKeys
19:44:12 error Jwt: One or more errors occurred.
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at VFCloudWebFront.JwtAuthModule.DownSignKeys()
https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
19:44:12 fail check download https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
. . .

19:45:33 error Jwt: One or more errors occurred.
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at VFCloudWebFront.JwtAuthModule.DownSignKeys()
https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
19:45:33 fail check download https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
The remote server returned an error: (400) Bad Request.
. . .

19:45:39 error Jwt: One or more errors occurred.
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at VFCloudWebFront.JwtAuthModule.DownSignKeys()
https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
19:45:39 fail check download https://login.microsoftonline.com/87e92c59-3863-486a-b08d-92b00cfe7264/v2.0/.well-known/openid-configuration
The remote server returned an error: (400) Bad Request.

Answer 2

I already found the problem, and it is with TLS, the solution was to put

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;  

This is at the thread level since if it is put only once in the startup of the application, it continues to give errors

Answer 3

Thanks for sharing the solution! I actually just navigated to this thread to suggest the TLS 1.2 enablement update as a possible cause, but it looks like you beat me to it. :) Another thing to note is that the .NET framework also needs to be updated to version 4.6.2 or later to be able to support TLS 1.2.

Enable TLS 1.2 Support