Hive Data Connection

DRK 1

We have a Hive datastore onprem which we are trying to connect via Java program hosted on Azure Batch. We have a self hosted integrated runtime to the onprem hive datastore. I would like to know how we can connect to this onprem site using java in Azure Batch program.

MartinJaffer-MSFT 26,026 Reputation points

2022-10-18T19:38:27.82+00:00

Hello and welcome to Microsoft Q&A @DRK

I understand you want to use Azure Batch to an on-prem Hive datastore. However, I'm confused about the self hosted integration runtime (SHIR). I'm not an expert in Azure Batch, but I don't think SHIR is a part of Batch. I am aware of the SHIR in the context of Azure Data Factory, Azure Synapse Analytics, Power BI, and maybe Logic Apps. Is one of these also involved?
DRK 1 Reputation point

2022-10-19T00:41:47.017+00:00

Yeah ok, I will try to rephrase it as I was not clear earlier. I am trying to connect to the on premises Hive data store from Azure batch. So the azure batch program (which is ConnectToHive.jar) need to connect to on premises hive. But I am not sure what my URL should be to connect to the on prem hive datastore from cloud. Hope that make a bit more sense.
DRK 1 Reputation point

2022-10-19T00:46:38.123+00:00

Sorry did not clarify the SHIR part. The SHIR was stood up to connect from ADF to on prem Hive store. I was thinking of leveraging that ADF connectivity in the java batch program. But that was my attempt to connect to on prem Hive. I was also not sure how to do that if that is even a possible route.
MartinJaffer-MSFT 26,026 Reputation points

2022-10-19T16:59:44.183+00:00

Well, Data Factory does have a Hive connector, and in the linked service you can specify the self hosted integration runtime. I think that might be easier than going directly from the Azure Batch. By specifying the SHIR in the ADF Hive linked service, you get access to the on-prem without having to open ports as you would for Azure Batch.

I assume your on-prem network has a firewall or private network preventing normal internet from accessing your hive store.
@DRK

Otherwise you would probably have to do some network stuff, like peer a VNet in Azure to your on-prem network. Then put the Batch pool in the Vnet so it can have native access.

Excerpt from batch-virtual-network:

To allow compute nodes to communicate securely with other virtual machines, or with an on-premises network, you can provision the pool in a subnet of an Azure VNet.
DRK 1 Reputation point

2022-10-19T20:31:43.937+00:00

Thanks Martin. The option you mentioned of using ADF with a linked service by specifying the SHIR is what I have tested and it is working as you said. But I am trying to use an existing jar file without rewriting, which is connecting to on-prem hive and doing a bunch of calculations. So I am looking for a way for the java program to communicate with the on-prem hive datastore via SHIR. Is it possible to do that ?
MartinJaffer-MSFT 26,026 Reputation points

2022-11-09T21:47:49.453+00:00

The SHIR is meant to extend ADF data movement. Data movement involves an ADF-owned compute resource, AKA Integration Runtime. The Batch is a compute resource not owned by ADF.

ADF can move data from on-prem to the Batch, or the output of the Batch to on-prem. However it does not extend the network reach of Batch.

I think @Xin Xin has the better idea.

1 answer

Xin Xin 26 Reputation points Microsoft Employee

2022-10-23T02:56:14.613+00:00

I think this might be a Hybrid-network issue, the task running Batch can connect anywhere if Vnet and NSG have done corresponding configuration. Commonly, we don't tent to expose such data transfer in Internet, so we can setup ExpressRoute or VPN between on-prem and Azure Cloud, route table, DNS resolve, then Batch program (ConnectToHive.jar) can connect on-prem Hive datastore like internal network.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment