Private endpoints for Azure Data Explorer
You can use private endpoints for your cluster to allow clients on a virtual network to securely access data over a private link. Private endpoints use private IP addresses from your virtual network address space to connect you privately to your cluster. Network traffic between clients on the virtual network and the cluster, traverses over the virtual network and a private link on the Microsoft backbone network, eliminating exposure from the public internet.
Using private endpoints for your cluster enables you to:
- Secure your cluster by configuring the firewall to block all connections on the public endpoint to the cluster.
- Increase security for the virtual network by enabling you to block exfiltration of data from the virtual network.
- Securely connect to clusters from on-premises networks that connect to the virtual network using a VPN gateway or ExpressRoutes with private-peering.
Overview
A private endpoint is a special network interface for an Azure service in your virtual network that is assigned IP addresses from the IP address range of your virtual network. When you create a private endpoint for your cluster, it provides secure connectivity between clients on your virtual network and your cluster. The connection between the private endpoint and the cluster uses a secure private link.
Applications in the virtual network can seamlessly connect to the cluster over the private endpoint. The connection strings and authorization mechanisms are the same as you'd use to connect to a public endpoint.
When you create a private endpoint for cluster in your virtual network, a consent request is sent for approval to the cluster owner. If the user requesting the creation of the private endpoint is also an owner of the cluster, the request is automatically approved. Cluster owners can manage consent requests and private endpoints for the cluster in the Azure portal, under Private endpoints.
You can secure your cluster to only accept connections from your virtual network by configuring the cluster firewall to deny access through its public endpoint by default. You don't need a firewall rule to allow traffic from a virtual network that has a private endpoint because the cluster firewall only controls access for the public endpoint. In contrast, private endpoints rely on the consent flow for granting subnets access to the cluster.
Plan the size of subnet in your virtual network
The size of the subnet used to host a private endpoint for a cluster can't be altered once the subnet is deployed. The private endpoint consumes multiple IP addresses in your virtual network. In extreme scenarios, such as high-end ingestion, the number of IP addresses consumed by the private endpoint might increase. This increase is caused by an increased number of transient storage accounts required as staging accounts for ingesting into your cluster. If the scenario is relevant in your environment, you must plan for it when determining the size for the subnet.
Note
The relevant ingestion scenarios that would be responsible for scaling out the transient storage accounts are ingestion from a local file and async ingestion from a blob.
Use the following information to help you determine the total number of IP addresses required by your private endpoint:
| Use | Number of IP addresses |
|---|---|
| Engine service | 1 |
| Data management service | 1 |
| Transient storage accounts | 6 |
| Azure reserved addresses | 5 |
| Total | 13 |
Note
The absolute minimum size for the subnet must be /28 (14 usable IP addresses). If you plan to create an Azure Data Explorer cluster for extreme ingestion workloads you are on the safe side with a /24 netmask.
If you created a subnet that is too small, you can delete it and create a new one with a larger address range. Once you've recreated the subnet, you can create a new private endpoint for the cluster.
Connect to a private endpoint
Clients on a virtual network using a private endpoint should use the same connection string for the cluster as clients connecting to a public endpoint. DNS resolution automatically routes connections from the virtual network to the cluster over a private link.
Important
Use the same connection string to connect to the cluster using private endpoints as you'd use to connect to a public endpoint. Don't connect to the cluster using its private link subdomain URL.
By default, Azure Data Explorer creates a private DNS zone attached to the virtual network with the necessary updates for the private endpoints. However, if you're using your own DNS server, you might need to make more changes to your DNS configuration.
Important
For optimal configuration, we recommend that you align your deployment with the recommendations in the Private Endpoint and DNS configuration at Scale Cloud Adoption Framework article. Use the information in the article to automate Private DNS entry creation using Azure Policies, making it easier to manage your deployment as you scale.
Azure Data Explorer creates multiple customer visible FQDNs as part of the private endpoint deployment. In addition to the query and ingestion FQDN it comes with several FQDNs for blob / table / queue endpoints (needed for ingestion scenarios)
Disable public access
To increase security, you also can disable public access to the cluster in the Azure portal.
Managed private endpoints
You can use a managed private endpoint to either enable the cluster to securely access your ingestion- or query-related services via their private endpoint. This allows the Azure Data Explorer cluster to access your resources via a private IP address.
Supported services
Azure Data Explorer supports creating managed private endpoints to the following services:
- Azure Event Hubs
- Azure IoT Hubs
- Azure Storage Account
- Azure Data Explorer
- Azure SQL
- Azure Digital Twins
Limitations
Private endpoints aren't supported for virtual network injected Azure Data Explorer clusters.
Implications on cost
Private endpoints or managed private endpoints are resources that incur additional costs. The cost varies depending on the selected solution architecture. For more information, see Azure Private Link pricing.