Accessing an ADLS Gen2 storage account using a private endpoint from a VM inside a VNet

Deborshi Nag 20 Reputation points
2023-08-22T09:38:13.2533333+00:00

I have a virtual machine inside a VNet with a Bastion. I have created an ADLS Gen2 storage with Hierarchical Namespace and connected using private endpoint.

What I experience now is that nslookup returns a private IP address from my subnet, which is good. I also can explore the storage container using Storage Explorer from inside the VM. However, I am unable to connect using Azure Portal from the virtual machine using a browser. Why is Azure Portal behaving differently from the Storage Explorer?

I tried the steps as per the documentation

https://learn.microsoft.com/en-us/azure/private-link/tutorial-private-endpoint-storage-portal?tabs=dynamic-ip

I was expecting to be able to connect to the storage container from the virtual machine, both using Storage Explorer and Azure Portal. I can connect using Storage Explorer, however Azure Portal fails.

Can anyone explain why Azure portal fails to connect?

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,185 questions
Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
511 questions
{count} votes

Accepted answer
  1. GitaraniSharma-MSFT 49,626 Reputation points Microsoft Employee
    2023-08-28T09:24:16.82+00:00

    Hello @Deborshi Nag ,

    I understand that you've a virtual machine inside a VNet with a Bastion and you created an ADLS Gen2 storage which is connected using private endpoint. You are able to explore the storage container using Storage Explorer from inside the VM, but you are unable to connect to storage container using Azure Portal from the virtual machine using a browser.

    This is expected behavior, when using Azure private endpoint.

    When using Storage Explorer from inside the VM, the storage explorer makes a direct connection using your Azure Vnet network to make requests to your storage resource privately.

    Refer: https://learn.microsoft.com/en-us/azure/storage/common/storage-explorer-network

    A private endpoint is a special network interface for an Azure service in your Virtual Network (VNet). When you create a private endpoint for your storage account, it provides secure connectivity between clients on your VNet and your storage. The private endpoint is assigned an IP address from the IP address range of your VNet. The connection between the private endpoint and the storage service uses a secure private link.

    And since the Storage Explorer from inside the VM uses the Azure Vnet network to make requests to your storage, it connects successfully.

    Refer: https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints#conceptual-overview

    But when accessing the storage using Azure Portal from the virtual machine via a browser, the connection is over Public Internet, and it will fail if public access to your Storage account is disabled.

    Refer: https://learn.microsoft.com/en-us/azure/private-link/tutorial-private-endpoint-storage-portal?tabs=dynamic-ip#disable-public-access-to-storage-account

    But you mentioned that you are able to connect to storage container using Azure Portal from the virtual machine using a browser if you use Azure service endpoint instead of Azure private endpoint.

    If you check the difference between Service Endpoint and Private endpoint, you can find the below:

    What is the difference between Service Endpoints and Private Endpoints?

    Private Endpoints grant network access to specific resources behind a given service providing granular segmentation. Traffic is sent privately using Microsoft backbone. It doesn’t traverse the internet. A Service Endpoint remains a publicly routable IP address. A Private Endpoint is a private IP in the address space of the virtual network where the private endpoint is configured. Refer: https://learn.microsoft.com/en-us/azure/private-link/private-link-faq#what-is-the-difference-between-service-endpoints-and-private-endpoints-

    After enabling a service endpoint, the source IP addresses switch from using public IPv4 addresses to using their private IPv4 address when communicating with the service from that subnet.

    With service endpoints, DNS entries for Azure services remain as-is today and continue to resolve to public IP addresses assigned to the Azure service. Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview#considerations

    Then you mentioned that it seems a bit conflicting because when you read about Service Endpoints, it says Service endpoints provide secure and direct connectivity to Azure services over the Azure backbone network which enable private IP addresses in the virtual network to reach an Azure service without the need of an outbound public IP.

    Yes, what you mentioned is true. But the meaning is a bit different.

    So, let's break it down to understand it better.

    Service endpoints allow you to secure your Azure resources to only your virtual networks but that doesn't explicitly mean private traffic only.

    When using service endpoint, the source IP address from where you connect to a service is taken into consideration.

    The public IP addresses for certain services are added to the route table by Azure when you enable a service endpoint to the service. Service endpoints are enabled for individual subnets within a virtual network, so the route is only added to the route table of a subnet a service endpoint is enabled for.

    Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#optional-default-routes

    Azure sends all traffic to the next hop type specified in the route, including traffic destined for public IP addresses of Azure services. When the next hop type for the route with the 0.0.0.0/0 address prefix is Internet, traffic from the subnet destined to the public IP addresses of Azure services never leaves Azure's backbone network, regardless of the Azure region the virtual network or Azure service resource exist in.

    Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#default-route

    If you check ID10 of the route table example in the below doc, you can find:

    https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview#route-tables

    Azure automatically added this route to the subnet when a service endpoint to an Azure service was enabled for the subnet. Azure routes traffic from the subnet to a public IP address of the service, over the Azure infrastructure network.

    So, when we say, "Virtual Network (VNet) service endpoint provides secure and direct connectivity to Azure services over an optimized route over the Azure backbone network", it means secure and direct connectivity to the Public IPs of Azure services from Virtual networks over the Azure backbone network. It is still public traffic but it doesn't go via public Internet (ISP network) but rather traverses through the Azure backbone network, which delivers your traffic over 165 edge Point of Presence (POP).

    To understand more about routing via Azure backbone network vs routing via public Internet (ISP network), you can refer the below doc:

    https://learn.microsoft.com/en-us/azure/virtual-network/ip-services/routing-preference-overview

    As shown in the screenshot of this document,

    User's image

    When a resource in Subnet B tries to reach any SQL Server, it uses a public IP address for outbound communication. The blue arrow represents this traffic. The SQL Server firewall must use that public IP address to allow or block the network traffic.

    When a resource in Subnet A tries to reach a database server, it's seen as a private IP address from within the virtual network. The green arrows represent this traffic. The SQL Server firewall can now specifically allow or block Subnet A. Knowledge of the public IP address of the source service is unneeded.

    Subnet A uses a service endpoint and hence an outbound public IP is NOT needed to reach the SQL server. It uses a private IP address in the Vnet to reach the SQL server, but it still connects to the Public IP of the SQL server.

    But when you use private endpoint, it creates a special network interface for an Azure service (such as SQL, Key Vault etc.) in your virtual network. The private endpoint is assigned an IP address from the IP address range of your virtual network. The connection between the private endpoint and the Azure service is a private link.

    As shown in the screenshot of this document,

    User's image

    The green arrow represents private link. A public IP address can still exist for the target resource alongside the private endpoint. The public IP is no longer used by the client application. The firewall can now disallow any access for that public IP address, making it accessible only over private endpoints.

    The client application typically uses a DNS host name to reach the target service. No changes are needed to the application. DNS resolution in the virtual network must be configured to resolve that same host name to the target resource's private IP address instead of the original public IP address. With a private path between the client and the target service, the client doesn't rely on the public IP address. The target service can turn off public access.

    So, in this case, any client from your subnet/Vnet can reach the SQL server as long as it can resolve the hostname to the private IP address of the private endpoint. There is no Public IP involved at all.

    To conclude:

    Service Endpoint is mainly a connection from a private IP address to a public IP address and traffic remains Public but goes over Azure backbone network.

    By enabling a service endpoint, it provides the identity of your virtual network to the Azure service. Once you enable service endpoints in your virtual network, you can add a virtual network rule to secure the Azure service resources to your virtual network.

    Refer: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-service-endpoints-overview#secure-azure-services-to-virtual-networks

    Client (subnet with service endpoint) ---> uses private IP address ------> to access the Public IP of the SQL server or any other Azure service.

    But Private endpoint is mainly a connection from a private IP address to a private IP address and the traffic is not public at all.

    By enabling a private endpoint, you're bringing the service into your virtual network.

    Client (subnet with private endpoint) ---> uses private IP address ------> to access the FQDN of the SQL server or any other Azure service which should resolve to the private endpoint IP address.

    Refer: https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview

    https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-dns#virtual-network-workloads-without-custom-dns-server

    • So, when enabling service endpoint on the subnet, you are just making sure the VMs in that subnet can use private IP address to access a Public Azure service. And hence you are able to access the Storage container using Azure Portal from the virtual machine via a browser.

    Refer: https://learn.microsoft.com/en-us/azure/virtual-network/tutorial-restrict-network-access-to-resources#restrict-network-access-to-a-subnet

    • But when you enable Private endpoint, the storage doesn't allow public access at all. And hence it fails, when accessing the Storage container using Azure Portal from the virtual machine via a browser because this connection is Public and private endpoint expects the client to connect to the Private IP address.

    Refer: https://learn.microsoft.com/en-us/azure/private-link/tutorial-private-endpoint-storage-portal?tabs=dynamic-ip#disable-public-access-to-storage-account

    Kindly let us know if the above helps or you need further assistance on this issue.


    Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.