Azure Pyton Pandas Automation into SQL Database

Adrian 181 Reputation points
2022-02-28T12:20:32.447+00:00

Hello! Looking to get some opinions on a process I am trying to incorporate.

I have a set of files being sent over/generated to me locally, I would like to upload these files to a container/dropbox, which would then trigger a Python Pandas Scripts (to transform the files) after which they would be save to a SQL Database on the Azure portal.

I don't mind that I would have to upload these files manually initially, what I am after is after this is done, the python script changes the files and saves it to a SQL Database automatically, every time a new file if uploaded to the initial location.

Any ideas/though very welcome.

Thanks!

Azure SQL Database
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,263 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,843 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions
Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,114 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Anurag Sharma 17,571 Reputation points
    2022-03-04T05:58:38.497+00:00

    Hi @Adrian , welcome to Microsoft Q&A forum.

    If we understand it correctly, you want to read the excel files from blob storage and load them into Azure SQL Databases.

    1) As mentioned by you, Azure Functions triggered by Blob storage will help us to achieve this requirements. Below article will help you with the same:

    Azure Blob storage trigger for Azure Functions

    2) If using the python script is not mandatory, we can use Azure Logic app as well which will read the files from Azure Blob Storage and load them into Azure SQL database:

    Azure Automation: Use Azure Logic Apps to import data into Azure SQL Database from Azure Blob Storage

    Again this logic app can be triggered if a blob is added/modified in storage:

    Create and manage blobs in Azure Blob Storage by using Azure Logic Apps

    3) Another way is use ADF to read the file from blob storage and trigger a python script.

    Tutorial: Run Python scripts through Azure Data Factory using Azure Batch

    However, out of all these I would suggest you to go with Azure function where we can fully use the existing python script. The article mentioned above will help you with technical details as well.

    Please let us know if this helps or else we can discuss further on the same.

    If answer is helpful please please click on 179888-image.png as it could help other members of the Microsoft Q&A community who have similar questions and are looking for solutions. Thank you for helping to improve Microsoft Q&A!

    0 comments