How to run Pymongo script in ADF

Question

How to run Pymongo script in ADF

Priyanka P 21

Hi,

We are trying to run pymongo code from ADF which will delete the data of a collection in Azure cosmosDB API for MongoDB.

please help with any documentation.

Priyanka P 21 Reputation points

2022-02-21T14:41:47.777+00:00

Hi Team,

Can we please get any response for this.

Accepted answer

0 additional answers

Your answer

Priyanka P 21 Reputation points

2022-02-21T14:41:47.777+00:00

Hi Team,

Can we please get any response for this.

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hello @Priyanka P ,

Thanks for the ask and using Microsoft Q&A platform .

PyMongo is a Python distribution containing tools for working with MongoDB, and is the recommended way to work with MongoDB from Python. So basically you are willing to run Python script from Azure data factory.

You can leverage custom activity in Azure data factory to perform this task. Using Azure batch services customer activity can run your script.

You can also deploy your python code to Azure functions and run Azure function using Azure Function activity.

As requirement is to run python from ADF, custom activity running in ADF will be better option.

Below documentation has detailed steps to Run Python Scripts through Azure data factory using Azure batch. Kindly go through same.
https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory

Please do let me know how it goes.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Priyanka P 21 Reputation points

2022-02-16T14:49:21.43+00:00

Hi Maheer,

Thanks for the quick reply.

Our requirement is to manipulate the collection data of Azure CosmosDB API for MongoDB, but this service has limitation from ADF so we are trying to do it with pymongo scripts to perform delete/update/truncate on cosmos collections.

As per the suggested documents tried this approach and below is the error details:

"Unable to cast object of type 'Microsoft.WindowsAzure.Storage.Blob.CloudAppendBlob' to type 'Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob'."

Below are some details of code:

command from ADF: "python abc.py"

code in the abc file :
import pymongo

myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["mydatabase"]
mycol = mydb["customers"]

mycol.drop()

abc.py was stored in AzureBlob

In the documents suggested to "Enable the start task and add the command cmd /c "pip install azure-storage-blob pandas", so we have updated as cmd /c "pip install azure-storage-blob python"

Please help if this is the right way to do.

Thanks
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-02-21T17:10:50.837+00:00

Hi @Priyanka P ,

For me its seems to be error with in code. That means Custom activity is able to run script. From error message it looks like Type cast issue.

Could you please try running code individually in local and see if its working. If its working locally and not here that means some issue with module installation may be. Also, kindly try having code and azure function and see if its working there? If it works there then we can use Azure function activity in ADF to run same. Thank you.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2022-02-27T17:23:42.697+00:00

Hi @Priyanka P - Just checking if you get chance to share updates. If this answer helped then please consider marking it as Accepted Answer. If you tried any other work around then please feel free to share here to community as well. Thank you.

Share via

How to run Pymongo script in ADF

Please do let me know how it goes.

0 additional answers

Your answer