Issue with Azure Privat Endpoint for Storage Account and F5 SSLVPN

Question

Issue with Azure Privat Endpoint for Storage Account and F5 SSLVPN

Jörg Lang 120

Hi All,

we are facing currently an issue with DNS Name Resolution, Azure Private Endpoint and SSLVPN by F5 which I would like to explian and I hope someone has an idea, how to solve it.

Following setup exists:

On Azure:

Azure Hub and Spoke
Private DNS Resolver
Storage Account with Private Endpoint
Private DNS Zone "privatelink.blob.core.windows.net" with an A-Entry for the Private Endpoint IP of Stoagreaccount

On Prem:

Conditional Forwarder for "blob.core.windows.net" to the dns inbound ip of Private DNS resolver

Testcase 1:

Nslookup from Server/VDI in DC network, "somename.blob.core.windows.net" is correctly resolved to IP Adress of the private endpoint, so everything works as expected.
Connection works

Testcase 2:

Same test, just from SSLVPN-Connected client: public IP is getting resolved and
no connection is possible because public access not allowed.

We have made some deeper investigation and identified the root cause.

The F5 ist using https://learn.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide to exclude O365 traffic from the VPN.

But in the O365 IP/Domain List, there is also following entry in

  {
    "id": 67,
    "serviceArea": "Common",
    "serviceAreaDisplayName": "Microsoft 365 Common and Office Online",
    "urls": [
      "*.blob.core.windows.net"
    ],
    "tcpPorts": "443",
    "expressRoute": false,
    "category": "Default",
    "required": false,
    "notes": "Security and Compliance Center eDiscovery export"
  }

This explains, why the resolution and traffic to a Storage Account is not going to the vpn connection.

So any idea how we can:

ignore/remove entry ID 67 on this web service list?

if not

force F5 to still route the storage account traffic to the VPN

Many thanks for your thoughts.

Regards

Joerg

KapilAnanth-MSFT 49,616 Reputation points Microsoft Employee Moderator

2023-11-10T09:25:46.3733333+00:00
@Jörg Lang

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

I understand that you are using a 3rd Party VPN solution to connect to Azure VNET from remote devices.

Please let me know if the above is incorrect.

Looking at your scenario,

I wouldn't recommend removing the required rule from O365 IP/Domain List

This would lead to issues with O365 services.

With respect to F5 configuration, I will recommend you to reach out to the vendor or related forums to get more details on this.

What I can recommend is to use Azure P2S VPN Client.

The idea here is to, forward the DNS requests for "*.blob.core.windows.net" traffic to the DNS server in Azure (Private DNS Resolver in your case)

Let it handle the DNS resolution and you will either get a Private IP or a Public IP

In case of Private IP, the traffic would be sent to the P2S VPN Tunnel if the Private IP belongs to the VNET.

However, in case of Public IP, the traffic would go to Internet.

Refer to Azure VPN Client DNS configuration for configuring DNS suffixes and custom DNS servers.

However, I agree that the above solution may not be the one you are looking and it will be challenging to provide any suggestions without understanding your other requirements and use cases (such as sticking to a 3rd party VPN Solution)

Kindly let us know if this helps or you need further assistance on this issue.

Thanks,

Kapil
Jörg Lang 120 Reputation points

2023-11-10T09:39:35.58+00:00
Hi @KapilAnanth-MSFT ,

thank you for your answer.

I understand that you are using a 3rd Party VPN solution to connect to Azure VNET from remote devices.

no, this is not the case.

Azure is connected by Express Route Circuit to DataCenter.

some Clients

are using VDI Desktops, running inside DC (connection/name resolution is working)

are connected by SSL VPN to DC (F5 System) where connection/name resolution isn't working

The idea here is to, forward the DNS requests for "*.blob.core.windows.net" traffic to the DNS server in Azure (Private DNS Resolver in your case)

Fully agree, but the mentioned entry with id 67 from the https://learn.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide list is avoiding that the request is pushed into the VPN tunnel and so to the DC DNS Servers where the conditional forwarder is configured to send the request to the Privat DNS Resolover to get the right IP.

Hope that clarifies the situation.

Regards

Jörg
KapilAnanth-MSFT 49,616 Reputation points Microsoft Employee Moderator

2023-11-10T12:08:59.0766667+00:00
@Jörg Lang

Thanks for the info.

In any case, we should implement a mechanism you must configure so that the traffic to "*.blob.core.windows.net" goes to the Private DNS Zone,

or

As you said, atleast force "<YourServiceName.blob.core.windows.net>" towards the Private DNS Zone (via DC).

I'd suggest you to check this with the vendor team if the above can be achieved.

In the document, I am not able to find any statement that explicitly states we should not route this traffic via a VPN.

The document states that you inspect and allow traffic to the endPoints/URLs specified here.

However, to get more details, I would suggest you file a support ticket where a support engineer can have a specialized 1:1 screen share session to investigate your environment.

In case you do not have a support plan, please do let us know, we will try and help you get a one-time free technical support.

Cheers,

Kapil
Jörg Lang 120 Reputation points

2023-11-10T13:20:03.2833333+00:00
@KapilAnanth-MSFT

Thanks again for the feedack.

From my perspective, it's a big mess that this MS WebService is publishing this wide range rule

{ "id": 67, "serviceArea": "Common", "serviceAreaDisplayName": "Microsoft 365 Common and Office Online", "urls": [ "*.blob.core.windows.net" ], "tcpPorts": "443", "expressRoute": false, "category": "Default", "required": false, "notes": "Security and Compliance Center eDiscovery export" }

These file of the MS webservice is used for many VPN-Solutions to configure split-traffic on the VPN where the list is maintaind automatically by MS. Exmaple please see here https://techdocs.f5.com/en-us/bigip-16-1-0/big-ip-access-policy-manager-network-access/configuring-address-spaces/creating-an-address-space-office.html

This "Rule" leads to the situation that any customer who want's/need's to have "non-public storage accounts" in combination with split-traffic for VPN has a problem with accessing private endpoints of storage accounts.

The "easiest" option, to sort that out, would be that on the webservice the customer has to option to filter out specific rules so that they are not fetched for split-tunnel configuration.

We have reached out to F5 support to see, if there is a solution available on F5 to "overrule" this specific rule anyhow, but to be honest this issue is created by MS and not by F5 or any other VPN-Solution vendor.

Also, as mentioned, the setup in the DC is completeley working, it's just an issue on "bad ms rule" and split-tunneling on VPN.

Regards

Joerg
KapilAnanth-MSFT 49,616 Reputation points Microsoft Employee Moderator

2023-11-10T13:30:36.2966667+00:00

@Jörg Lang

Wrt,

This "Rule" leads to the situation that any customer who want's/need's to have "non-public storage accounts" in combination with split-traffic for VPN has a problem with accessing private endpoints of storage accounts.

I agree.

I am afraid my expertise is over Azure and have limited knowledge on O365 Products and it's requirements.

I have added the "Microsoft 365" tag so that the community experts can add their comments.

Meanwhile, you can open a Support ticket with Microsoft and understand if the configuration file can be fine tuned for the use case.

Cheers,

Kapil