Can't connect to HBase instance from Azure Data Factory to use it as Source

Question

Can't connect to HBase instance from Azure Data Factory to use it as Source

mateusz.peczek 16

I'm trying to connect to Apache HBase instance (self hosted, not HDInsight from Azure) and I'm out of ideas how to do that. I've already tested few different options.

I've created Azure VM and installed HBase instance there. I've opened required ports to enable communication from outside of this VM. All necessary requests are working calling them from my local PC.

rules

Approaches that I've taken are:

Linked service based on built in HBase connector pointed to Thrift service.

I've set up a Thrift server on HBase instance hosted on Azure VM. I'm able to see REST control panel for this service from outside. Here's config window:

thrift

This returns error:
ERROR [HY000] [Microsoft][DriverSupport] (1110) Unexpected response received from server. Please ensure the server host and port specified for the connection are correct and confirm if SSL should be enabled for the connection.

Linked service based on built in HBase connector pointed to REST API service.

rest

This configuration works from LinkedService point of view. After creating DataSet, all the tables are listed too, but no data can be retrieved.

data from rest

Error from Preview Data options is:

ERROR [HY000] [Microsoft][HBase] (40) Error with HTTP request, response code: 404

Linked service based on ODBC driver and self hosted integration runtime.

As last resort option I've installed self hosted integration on my machine and CData ODBC Driver for Apache HBase. I'm able to retrieve all the tables and data by driver itself:

odbc driver

Both Self Hosted IR and ODBC linked service based on that IR works as expected, showing successful connection status

integration

In that case, Data Set cannot list any of the tables that exists in HBase cluster.

dataset

Error received after selecting Preview data:
ERROR [42000] Invalid URI: The hostname could not be parsed.

I've tried multiple ways of providing Host address in connection string, using http:// with IP address and IP address alone, for each case result is the same. I've tried to use known value "test" as table name using Edit option, but this one didn't work too.

I have simple C# application with Thrift client inside and it works without any issues on port 9090, so it's seems not related to networking or authorization (which is disabled for my instance). This suggests, that issue is only on HBase or ODBC connector used by ADF, since alternatives ran on my local machine are working without any issues.

Was any of you able to set working connection to self hosted HBase as sink using ADF?

ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator

2023-06-22T10:23:22.06+00:00

Hi mateusz.peczek,

Thank you for posting query in Microsoft Q&A Platform.

I think since your liked service is based on REST API hence it's listing tables alone but not able to fetch data.

Regarding, connection from HBase linked service, it seems SSL is not enabled or some host name and port or incorrect or some network related issue.

Are we sure is network configurations and Firewall settings all are correct in HBase? Kindly recheck and make sure no network connectivity issues between HBase and Azure Data Factory.

Also, Did you tried with Selfhosted IR directly in HBase connector for creating linked service? If no, kindly try that once.
Please let me know how it goes. Thank you.
mateusz.peczek 16 Reputation points

2023-06-22T11:50:42.4366667+00:00

Hello , @ShaikMaheer-MSFT , thank you for your response :)

I have both options active on HBase instance - REST API and Thrift service.

I don't think that SSL or any network connection is the issue here, since I am able to establish connection from ADF to list the tables without it and it's not enabled at all on my Virtual Machine that hosts it. As I mentioned, I have simple C# application on my local PC that is able to connect to VM's HBase instance without any problems using Thrift service (port 9090). Using Postman app on my PC to call REST service (port 8080) works as well without any SSL authorization added.

I've tried to recreate the same type of linked services on my Self Hosted IR and I'm getting exactly the same errors as on Azure Hosted one. I did that using "Others" option since there's no HBase option there and copied the JSON definition from existing linked services in Data Factory.

It seems very close to work properly, since in case when linked service is pointed to port 8080 (using REST), linked service is able to retrieve all tables, but not the data in it. It returns
ERROR [HY000] [Microsoft][HBase] (40) Error with HTTP request, response code: 404
when Preview data is chosen.
mateusz.peczek 16 Reputation points

2023-06-23T14:51:04.71+00:00

I've recreated whole setup again, this time using HDinsight cluster with HBase database.

I've got to the same step, where Linked Service is able to connect to database, list the tables but not retrieve any data.

Here's list of commands I've used to populate database with mock data.

Here's example of visible table when creating new data set based on this linked service with error on previewing the content of the table:
ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator

2023-06-28T06:19:05.8433333+00:00

Hi mateusz.peczek, Looks strange. I would encourage you to have support ticket for this for deeper investigation and understanding. if you already found the solution, kindly feel free to share with community as well. That helps others too. Thank you.
mateusz.peczek 16 Reputation points

2023-06-29T11:14:47.26+00:00

Hi @ShaikMaheer-MSFT , unfortunatelyI haven't found solution, I've tried to experiment with various values and configurations and the furthest I went was to list the tables, but have that 404 error on receiving data as described.
navaneeth kishen 0 Reputation points

2023-09-10T08:00:52.6366667+00:00

Hi @mateusz.peczek . I am facing the same issue. Linked Service is able to connect to the database, and list the tables but not retrieve any data. And the pipeline fails as well. I would like to know if you have already found any solution. Thank you.

Your answer

ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator

2023-06-22T10:23:22.06+00:00

Hi mateusz.peczek,

Thank you for posting query in Microsoft Q&A Platform.

I think since your liked service is based on REST API hence it's listing tables alone but not able to fetch data.

Regarding, connection from HBase linked service, it seems SSL is not enabled or some host name and port or incorrect or some network related issue.

Are we sure is network configurations and Firewall settings all are correct in HBase? Kindly recheck and make sure no network connectivity issues between HBase and Azure Data Factory.

Also, Did you tried with Selfhosted IR directly in HBase connector for creating linked service? If no, kindly try that once.
Please let me know how it goes. Thank you.
mateusz.peczek 16 Reputation points

2023-06-22T11:50:42.4366667+00:00

Hello , @ShaikMaheer-MSFT , thank you for your response :)

I have both options active on HBase instance - REST API and Thrift service.

I don't think that SSL or any network connection is the issue here, since I am able to establish connection from ADF to list the tables without it and it's not enabled at all on my Virtual Machine that hosts it. As I mentioned, I have simple C# application on my local PC that is able to connect to VM's HBase instance without any problems using Thrift service (port 9090). Using Postman app on my PC to call REST service (port 8080) works as well without any SSL authorization added.

I've tried to recreate the same type of linked services on my Self Hosted IR and I'm getting exactly the same errors as on Azure Hosted one. I did that using "Others" option since there's no HBase option there and copied the JSON definition from existing linked services in Data Factory.

It seems very close to work properly, since in case when linked service is pointed to port 8080 (using REST), linked service is able to retrieve all tables, but not the data in it. It returns
ERROR [HY000] [Microsoft][HBase] (40) Error with HTTP request, response code: 404
when Preview data is chosen.
mateusz.peczek 16 Reputation points

2023-06-23T14:51:04.71+00:00

I've recreated whole setup again, this time using HDinsight cluster with HBase database.

I've got to the same step, where Linked Service is able to connect to database, list the tables but not retrieve any data.

Here's list of commands I've used to populate database with mock data.

Here's example of visible table when creating new data set based on this linked service with error on previewing the content of the table:
ShaikMaheer-MSFT 38,531 Reputation points Microsoft Employee Moderator

2023-06-28T06:19:05.8433333+00:00

Hi mateusz.peczek, Looks strange. I would encourage you to have support ticket for this for deeper investigation and understanding. if you already found the solution, kindly feel free to share with community as well. That helps others too. Thank you.
mateusz.peczek 16 Reputation points

2023-06-29T11:14:47.26+00:00

Hi @ShaikMaheer-MSFT , unfortunatelyI haven't found solution, I've tried to experiment with various values and configurations and the furthest I went was to list the tables, but have that 404 error on receiving data as described.
navaneeth kishen 0 Reputation points

2023-09-10T08:00:52.6366667+00:00

Hi @mateusz.peczek . I am facing the same issue. Linked Service is able to connect to the database, and list the tables but not retrieve any data. And the pipeline fails as well. I would like to know if you have already found any solution. Thank you.

Share via

Can't connect to HBase instance from Azure Data Factory to use it as Source

Your answer