How to generate ETL pipeline for Python code which gives 2 JSON file in response

Question

Requirement :

I have python code in my local pc folder which returns 2 JSON file as a output (that code is using API and try to store the data of website in JSON file),

I want to create automation for running that python code at particular time using ADF pipeline so that I don't need to run the code manually in my local PC manually .

Please help me in creating the pipeline for the same I need detailed explanation how will I achieve the following requirement ( I am new to ADF) .

Answer 1

Hi Utsav Mori,

I would suggest you to start with the quick starter on Data Factory (https://learn.microsoft.com/en-us/azure/data-factory/quickstart-get-started) then you can set up a pipeline base on your requeriments . It would look like so:

1. Create an Azure Batch Account and Pool.
2. Upload your Python script to Azure Blob Storage.
3. Create a Data Factory and a new pipeline in it.
4. Add a Custom Activity in the pipeline to run your Python script.
5. Define the input and output datasets linked to your Blob Storage.
6. Validate and publish the pipeline.
7. Create a schedule trigger for automation.
8. Monitor the pipeline’s performance.

Additional Microsoft how tos:

• https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-schedule-trigger?tabs=data-factory
• https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal

If the information helped address your question, please Accept the answer.

Luis

Answer 2

Hi @Utsav Mori
Thank you for your question and using the Microsoft Q&A platform

To run a Python script in Azure Data Factory (ADF), you can use the following approaches:

1. Azure Batch: You can use Azure Batch to run your Python script in parallel on multiple virtual machines (VMs) to improve performance. This approach is suitable for large-scale data processing.
2. Azure Functions: You can use Azure Functions to run your Python script as a serverless function. This approach is suitable for small-scale data processing.
3. Azure Databricks: You can use Azure Databricks to run your Python script in a distributed environment. This approach is suitable for large-scale data processing and machine learning workloads.

For more information, please refer the below articles:

• https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory
• https://learn.microsoft.com/en-us/azure/data-factory/control-flow-azure-function-activity
• https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-python

It is possible to run a Python script in Azure Data Factory (ADF) without creating a Batch account and pool. You may checkout the below link to achieve the goal of automating the Python script execution and storing the JSON output using ADF with Azure Functions:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-function-app-portal?pivots=programming-language-python#create-a-function-app

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.