Azure Firewall TLS inspection fails with handshake failure, alert 40

MATEUSZ U 0

interCA-old.pfx.txt

Hello, I'm trying to setup Azure Firewall with TLS inspection. I cannot get past one problem. Problem: Firewall fails to process rule. Chrome/Edge browser error: ERR_SSL_VERSION_OR_CIPHER_MISMATCH openssl error:

$ openssl s_client -connect app-service-xyz.ase-xyz.appserviceenvironment.net:443 -servername app-service-xyz.ase-xyz.appserviceenvironment.net
CONNECTED(000001D4)
C0140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../openssl-3.1.4/ssl/record/rec_layer_s3.c:1586:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 386 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

My setup: Azure Firewall premium Azure Firewall policy (premium) Firewall access to Key vault: managed identity with full access (for testing purposes) IDPS: Alert mode Network setup: Two subnets, VM in subnet A, private endpoints to ASE App service in subnet B, traffic from one subnet to the other routed through firewall which is in another vnet which is peered (simulation of hub&spoke model with WAF+Firewall security stack). Application rule: From "10.1.4.0/24" (subnet A) to FQDN/URL "*.ase-xyz.appserviceenvironment.net" with TLS inspection=Yes and protocol: "Https:443". Flipping TLS Inspection to "No" makes website work correctly when loaded from VM but without certificate substitution. CA certificate: tried multiple times changing different params based on bash instructions in https://learn.microsoft.com/en-us/azure/firewall/premium-certificates, tried adding it as a secret or as a certificate to keyvault. Nothing worked. The basic certificate created according to instruction is in the attachement. Logs: Log workspace does not report any rule attempt (be it Allow or Deny) as if it was failing so early that no logging is possible yet.

MATEUSZ U 0 Reputation points

2024-02-17T00:40:42.8533333+00:00

Example of certificate used: interCA-old.pfx.txt interCA-old2.pfx.txt interCA-old1.pfx.txt interCA.pfx.txt
KapilAnanth-MSFT 46,676 Reputation points Microsoft Employee

2024-02-19T05:49:35.2733333+00:00
@MATEUSZ U ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you would like to do a POC for Azure Firewall TLS inspection.

I see you are testing East-West TLS Inspection as the destination is a Private EndPoint.

As next steps,

Can you confirm if the Root CA certificate is installed on client operating system.

As this is testing, you can also consider using the Certificate auto-generation feature.

If you do so, make sure you download the certificate in .CER format and copy it to the end user’s machine.

See Blog : Building a POC for TLS inspection in Azure Firewall

Can you run the openssl command with "-debug" parameter and share the results?

Cheers,

Kapil
Mateusz U 61 Reputation points

2024-02-19T13:09:35.7733333+00:00
@KapilAnanth-MSFT Thank you for attention. What I'm trying is indeed east-west inspection of traffic coming from WAF but WAF is also not working, backend pool shows in red (Cannot connect to backend server. Check whether any NSG/UDR/Firewall is blocking access to the server. Check if application is running on correct port. To learn more visit - https://aka.ms/servernotreachable.) so what I'm trying to get working first is connecting from VM on which I can at least start debugging what's wrong. Also some additional info: location is West Europe. I tried as you instructed:

I used auto-generation feature which created such certificate: fw-cert-R5NTgc3CDer4a_9fef53cd807b467c8343e4f3b819cf5e.cer.txt (take notice that the cert is 2048 bits which by default seems wrong)

I confirmed that Root certificate is installed on the machine (actually it seems auto-generated certificate is root cert and is used in Azure Firewall directly to sign others). I restarted machine after installation just to be sure.

Openssl with -debug flag on autogenerated certificate below: openssl_result.txt

openssl s_client -connect sandbox-psd2sandbox-webapp13.test-example-asev3434-sandbox.appserviceenvironment.net:443 -servername sandbox-psd2sandbox-webapp13.test-example-asev3434-sandbox.appserviceenvironment.net -debug

CONNECTED(000001B4) write to 0x1b910dc48b0 [0x1b9111fc630] (386 bytes => 386 (0x182)) 0000 - 16 03 01 01 7d 01 00 01-79 03 03 2c 03 ee 44 fe ....}...y..,..D. 0010 - 6a 15 29 84 08 7e de d5-a0 9a 2d cb 9b 46 2f 7a j.)..~....-..F/z 0020 - 2b 70 df 6c 50 3e 30 0f-f6 60 23 20 48 47 3b 4a +p.lP>0..`# HG;J 0030 - ae f5 7c c5 ad e1 e1 40-9b df d5 de bd 86 c1 37 ..|....@.......7 0040 - 27 ac e9 04 19 11 cb be-4c f8 54 0e 00 3e 13 02 '.......L.T..>.. 0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa .....,.0........ 0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27 .+./...$.(.k.#.' 0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d .g.....9.....3.. 0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 f2 ...=.<.5./...... 0090 - 00 00 00 59 00 57 00 00-54 73 61 6e 64 62 6f 78 ...Y.W..Tsandbox 00a0 - 2d 70 73 64 32 73 61 6e-64 62 6f 78 2d 77 65 62 -psd2sandbox-web 00b0 - 61 70 70 31 33 2e 74 65-73 74 2d 65 78 61 6d 70 app13.test-examp 00c0 - 6c 65 2d 61 73 65 76 33-34 33 34 2d 73 61 6e 64 le-asev3434-sand 00d0 - 62 6f 78 2e 61 70 70 73-65 72 76 69 63 65 65 6e box.appserviceen 00e0 - 76 69 72 6f 6e 6d 65 6e-74 2e 6e 65 74 00 0b 00 vironment.net... 00f0 - 04 03 00 01 02 00 0a 00-16 00 14 00 1d 00 17 00 ................ 0100 - 1e 00 19 00 18 01 00 01-01 01 02 01 03 01 04 00 ................ 0110 - 23 00 00 00 16 00 00 00-17 00 00 00 0d 00 2a 00 #.............*. 0120 - 28 04 03 05 03 06 03 08-07 08 08 08 09 08 0a 08 (............... 0130 - 0b 08 04 08 05 08 06 04-01 05 01 06 01 03 03 03 ................ 0140 - 01 03 02 04 02 05 02 06-02 00 2b 00 05 04 03 04 ..........+..... 0150 - 03 03 00 2d 00 02 01 01-00 33 00 26 00 24 00 1d ...-.....3.&.$.. 0160 - 00 20 2e 79 f5 58 ea 3e-dd b5 da 24 67 bc 9c 4a . .y.X.>...$g..J 0170 - b4 5b 52 a4 91 1a 42 1a-e3 12 b4 6c 5a 56 5b 26 .[R...B....lZV[& 0180 - e5 21 .! read from 0x1b910dc48b0 [0x1b9111ef373] (5 bytes => 5 (0x5)) 0000 - 15 03 03 00 02 ..... read from 0x1b910dc48b0 [0x1b9111ef378] (2 bytes => 2 (0x2)) 0000 - 02 28 .( AC1D0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../openssl-3.1.4/ssl/record/rec_layer_s3.c:1586:SSL alert number 40 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 386 bytes Verification: OK --- New, (NONE), Cipher is (NONE) This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- read from 0x1b910dc48b0 [0x1b910d757d0] (8192 bytes => 0)
KapilAnanth-MSFT 46,676 Reputation points Microsoft Employee

2024-02-20T10:12:17.02+00:00
@MATEUSZ U ,

Can you share an architecture diagram of your set up?

Please include the UDR you defined for the App Gateway's subnet

Where exactly is WAF in your architecture?

I believe this should be Application Gateway in front of Azure Firewall , but please confirm.

The error you shared, "https://aka.ms/servernotreachable" , indicates there is a misconfiguration with respect to App Gateway's subnet.

If you are using UDRs, you must also make sure the return traffic from your backend is able to reach application gateway.

See the Architecture for Application Gateway before Firewall (highlighted)

Moreover, I don't think using a Self-signed certificate will work in this scenario.

You are doing End-to-End TLS.

Application Gateway terminates the TLS sessions at the gateway and decrypts user traffic. Application Gateway then initiates a new TLS connection to the backend server and re-encrypts data using the backend server's public key certificate before transmitting the request to the backend

This means, your VM is reaching out to the App gateway and thus, will expect App Gateway to have a valid certificate (not self-signed) for whatever origin is hosted.

In your case, it should be "app-service-xyz.ase-xyz.appserviceenvironment.net"

And this is not interacting with the certificate created for AzFW.

For POC,

I would suggest remove Azure App gateway and directly try the set-up using Azure Firewall with TLS inspection - Make sure this works.

Then, going forward, you can use a Enterprise CA certificates for Production workloads -which the App gateway will trust.

Cheers,

Kapil
Mateusz U 61 Reputation points

2024-02-22T16:07:36.3566667+00:00
Sorry I kept you waiting. Here is the diagram(simplified one) and terraform code if you'd like to try it yourself. diagram.png

terraform.zip.txt Also in the meantime of processing this issue I changed the design a bit - extracted WAF into a separate VNet so that WAF is forced to go through firewall to reach applications. But the result is neverless the same. The diagnostics i did was from the vm in a diagnostic subnet. EDIT: note that after changing terminate_tls to false it starts working.

terminate_tls = false
Mateusz U 61 Reputation points

2024-02-23T16:05:57.1533333+00:00
@KapilAnanth-MSFT Additionally to answer the other questions:

I think everything from networking perspective is set up correctly regarding routing rules (I mean UDR/NSG/Firewall) because with terminate_tls = false it works. Do you think my assumption is right ?

Regarding "I would suggest remove Azure App gateway and directly try the set-up using Azure Firewall with TLS inspection - Make sure this works." - as you'll be able to see in details in terraform code the "diagnostics vm" is trying to directly reach app service through Firewall without going through Application gateway. That's how I've been able to at least get those curl results for you. To summarise

Trying VM -> Firewall -> App service private endpoint results in

ERR_SSL_VERSION_OR_CIPHER_MISMATCH in browser

Alert 40 in curl

Trying Public internet -> Application Gateway (with WAF) -> Firewall -> App service results in 502 Bad Gateway with "Cannot connect to backend server. Check whether any NSG/UDR/Firewall is blocking access to the server. Check if application is running on correct port. To learn more visit - https://aka.ms/servernotreachable." as error in backend pool health check.

I was trying actually a few options here - setting or not setting my fake Root CA to be trusted in backend setting but nothing worked.

Regarding "Then, going forward, you can use a Enterprise CA certificates for Production workloads -which the App gateway will trust." - that's exactly what I'm planning but I wanted PoC to try first. But I can try to get real CA in my enterprise to check if it works. It will just take a while for me because it's longer path to get it.
KapilAnanth-MSFT 46,676 Reputation points Microsoft Employee

2024-02-25T05:42:25.1866667+00:00

@MATEUSZ U ,

Thanks for the detailed summary.

First, we must make sure Scenario "a" works (TLS inspection with a self-signed cert) . Then we can proceed to "b".

However, to troubleshoot thoroughly, I believe we will need a specialized 1:1 session, where a support engineer can have a screen share session and access to your subscription. If you have a support plan you may file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Mateusz U 61 Reputation points

2024-02-29T12:23:17.44+00:00

thank you, I'm preparing the setup right on on official test environment in my company and if the error persists (which I think it will) I'm gonna create support ticket and post it here. If it does not persist I'll post here what I found out caused the issue since having working vs non-working should be easy to check.

Share via

Azure Firewall TLS inspection fails with handshake failure, alert 40

Your answer