Timeout in hive (hadoop) after swoitching to use private endpoint storage

Tal Ivshin 1

We have an aks cluster that has access to a storage account (we use managed identities to allow it to access the storage account)

The storage account has selected network access, we limit access from the aks node subnets. We have hive installed in the cluster and it is able to connect to the storage account.

When we disable the public network access (this is done via terraform) - we also create a private endpoint for the storage with an IP address from the same subnet as the aks nodes - hive is unable to access the storage account and we see timeout errors like the following:

9:45:40.260 [pool-6-thread-183] ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler - Retrying HMSHandler after 2000 ms (attempt 6 of 10) with error: javax.jdo.JDODataStoreException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

Prrudram-MSFT 21,886 Reputation points

2023-03-31T04:40:29.4633333+00:00
Hello @Tal Ivshin

You are facing a timeout error when you disable the public network access and create a private endpoint for the storage account with an IP address from the same subnet as the AKS nodes. This error indicates that the connection to the storage account is timing out.

One probable reason for this issue is that the private endpoint is not properly configured to allow traffic from the AKS nodes. You may need to check the configuration of the private endpoint and ensure that it is properly configured to allow traffic from the AKS nodes.

Another probable reason for this issue is that the firewall rules on the storage account are not properly configured to allow traffic from the private endpoint. You may need to check the firewall rules on the storage account and ensure that they are properly configured to allow traffic from the private endpoint.

You can also refer to the following documentation for more information on how to configure private endpoints for Azure Storage accounts:

Configure private endpoints for Azure Storage accounts

If you continue to see this issue, I recommend contacting technical support team for further debugging.
If you don't have the ability to open a support ticket, you can send an email with your Subscription ID and link to this thread to AzCommunity@microsoft.com with subject line as "ATTN: Priyanka". I can then enable your subscription for one-time free support.
Prrudram-MSFT 21,886 Reputation points

2023-04-05T18:56:47.7966667+00:00

@Tal Ivshin What is the progress here?
Sumarigo-MSFT 43,411 Reputation points Microsoft Employee

2023-04-08T01:44:11.5833333+00:00

@Tal Ivshin Just checking in to see if the above answer helped. If this answers your query, do click "Accept the answer” for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.