Use Function App to Ingest data from tables inside Azure Databrick

Question

I have a table "hive_metastore.hr.book_information" inside Azure Databricks. This table contains some columns which have nested structure.
I want to read data from this table using Azure Function (with a query to ingest exactly rows that i need). I expect that I can query the data and output data can keep the structure and the response time is low.
Is there any way to do this ?

Thank for your helping.

Answer 1

@Khanh Dang Viet

To ingest data from a table in Azure Databricks using an Azure Function, you can utilize the Azure Databricks REST API or ODBC to connect to your Databricks workspace. Here's a general approach:

Using PyODBC with an Azure Function (Python)

This approach connects to Databricks using ODBC and runs a SQL query.

Set Up an ODBC Connection - Install the ODBC driver. Download the Databricks ODBC Driver from Databricks Documentation.

Set up a Databricks SQL Warehouse (formerly called SQL Endpoint) in the Databricks UI.

Configure the ODBC connection string with:

• Databricks Server hostname
• HTTP Path
• Access Token (from Databricks UI)

Write Azure Function (Python) to Query Data. Create an HTTP-triggered Azure Function to fetch data.

Using the Databricks REST API

Generate a Databricks Token

Go to Databricks Workspace → User Settings → Access Tokens → Generate a token.

Call the API from an Azure Function.

Use the /api/2.0/sql/statements endpoint to execute queries.

Similar issues: https://learn.microsoft.com/en-us/answers/questions/1810630/what-is-the-best-way-to-access-data-in-the-data-br

https://learn.microsoft.com/en-us/answers/questions/723551/query-azure-databricks-table-from-azure-function-a

For more details refer: https://learn.microsoft.com/en-us/azure/databricks/tables/streaming#before-you-begin

https://learn.microsoft.com/en-us/azure/databricks/query/#what-data-can-you-query-with-azure-databricks

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.