question

ronkit-5872 avatar image
0 Votes"
ronkit-5872 asked vipullag-MSFT edited

Azure Key Vault Managed Identities SSL connection could not be established

Hi,

I have two Azure App Service running in Window Container Plan which push from ACR using User Managed Identity with Reader role for accessing Azure key Vault.
The first app is working fine, but the second is not working. They have the exactly same code as below which using .Net Core 3.1:

207955-image.png




After deploying the second App Service, I keep getting following error:

 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] - Unhandled exception. System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -  ---> Azure.RequestFailedException: The SSL connection could not be established, see inner exception.
 02/06/2022 09:26:26.115 WARNING - Site: sea-app-uat-dcoadmin-01 - Container producing too many logs. Suspending temporarily.
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -  ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -  ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] - --- End of stack trace from previous location where exception was thrown ---
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.ThrowIfExceptional()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] - --- End of stack trace from previous location where exception was thrown ---
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    --- End of inner exception stack trace ---
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    --- End of inner exception stack trace ---
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    --- End of inner exception stack trace ---
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Security.KeyVault.KeyVaultPipeline.GetPageAsync[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+MoveNext()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Core.PageResponseEnumerator.FuncAsyncPageable`1.AsPages(String continuationToken, Nullable`1 pageSizeHint)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.AsyncPageable`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.LoadAsync()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
 02/06/2022 09:26:26.115 STDERR - Site: sea-app-uat-dcoadmin-01 - [7ae78f9bc29451e8029f9caa7861ee2aa55aadba6fffa529a5f25dbd8f67d658] -    at Microsoft.Extensions.Hosting.HostBuilder.Build()

Anyone have any idea? Any help will be appreciate

Thanks
Ron

dotnet-aspnet-core-generalazure-managed-identitydotnet-aspnet-core-auth
image.png (143.7 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@ronkit-5872
Thank you for your post!

I'm not familiar with Azure Container Registries, but from your logs it looks like it could be related to an SSL issue. However, to gain a better understanding of your environment and to add the correct community tags, can you share any documentation that you're following?

 #From your logs, it looks like the app could be running into an SSL connection issue.
 02/06/2022 09:26:26.115 WARNING - Site: sea... - Container producing too many logs. Suspending temporarily.
 02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -  ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -  ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

 #Because of the SSL issue, there's a cancel/end auth log which could be causing the "Azure.Core.Pipeline.RetryPolicy.ProcessAsync" to be triggered.
 System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
  02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -    at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
  02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -    at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)

I hope this helps!


If you have any other questions, please let me know.
Thank you!

0 Votes 0 ·

Hi @JamesTran-MSFT , im referring this current doc
https://docs.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1
Section "Using Managed Identities for Azure Resources"

0 Votes 0 ·
JamesTran-MSFT avatar image
0 Votes"
JamesTran-MSFT answered JamesTran-MSFT edited

@ronkit-5872
Thank you for following up on this!

Based off the warnings and STDERR's within your logs, it doesn't look like this is directly an issue with the managed identity or Key Vault. However, this could be more related to the SSL connection as I mentioned earlier, so I've added the .NET community support tags to this thread so their experts can look into this issue as well.


  • Because you have two apps with the same config, and the first app is working as expected, do you know if the second app has any networking config that could be causing this issue?

  • Is the second app sharing the same resources (i.e. KV)?

Potential Issues:

  02/06/2022 09:26:26.115 WARNING - Site: sea... - Container producing too many logs. Suspending temporarily.
  02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -  ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
  02/06/2022 09:26:26.115 STDERR - Site: sea... - [7ae...] -  ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

When it comes the SSL messages, I found a few Stack Overflow threads that might help point you in the right direction.

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception:

System.IO.IOException: Authentication failed because the remote party has closed the transport stream:

  • System.IO.IOException: Authentication failed - This error is generally related to the security protocol type. One customer resolved their issue by force TLS 1.2 and compiling their app with the latest version of .NET.

  • Auth Failed because remote party has closed the transport stream - You may get this error when trying to call an external API. This error is related to the Security Protocol Type, it is most likely caused by your application's default security protocol type being set too low, A lot of external APIs now expect requests using TLS 1.2 or above.


I hope this helps!


If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.


Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @JamesTran-MSFT ,

Yea, I have also look through quite some online resources. Added the security protocol thingy as below too. FYI, I am using .Net Core 3.1

208344-image.png


208376-image.png


Both of the application is under the same solution
208412-image.png


  • Yes, the first application is working fine without any issue, only occurs on the second application. Suspecting something wrong with the second application, but so far still no idea on where it is

  • Yes, both application is sharing the same KV resource. Just wondering can Azure KV be shared among different solution?


Anyway, thanks for your insight. I will keep investigate on this, so far still no luck

Thanks
Ron

0 Votes 0 ·
image.png (64.4 KiB)
JasonPan-MSFT avatar image
1 Vote"
JasonPan-MSFT answered

Hi @ronkit-5872

The problem is when the program is released and restarted. So please use the diagnostic tool to check your App Service.

208598-image.png

You will find some useful information, if you need further help, please let me know.

Best Regards,
Jason



image.png (113.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.