How to use Python library in SQL platform without using Databricks?

Mohammad Saber 591 Reputation points
2023-10-03T21:29:14.32+00:00

Hi,

I have developed a Python library that is installed on a Databricks cluster (as a wheel file) and is used to apply a few functions on delta tables:

Import library into a notebook

Read required data from delta tables

  • Run python functions on the data and store results in a delta table

I need to run the same python codes in an Azure environment that doesn't have Databricks workspace. The client doesn't use Azure Databricks and all tables are SQL tables stored in Azure SQL database.

Is there any way or service that I can use the same python library in a SQL platform without using Databricks (e.g Data Factory, Azure Function)?

It is not possible to write all codes in SQL language so I need to use the python library.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
12,895 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Bruce (SqlWork.com) 57,891 Reputation points
    2023-10-04T17:59:40.4166667+00:00
    1 person found this answer helpful.

  2. QuantumCache 20,031 Reputation points
    2023-10-04T16:49:19.3733333+00:00

    Hello @mohammad saber

    Please see the below suggestion from Bruce on hosting python scripts.

    You can try [Azure Functions]() to run your Python library in a SQL platform without using Databricks. Azure Functions is a serverless compute service that allows you to run event-driven code in a variety of languages, including Python. By using Azure Functions to run your Python library, you can apply the required functions on the data stored in your Azure SQL database without using Databricks. You can trigger the function based on a schedule or an event, depending on your requirements. Quickstart: Create a function in Azure with Python using Visual Studio Code Please let us know if that helps?