How can I automatically delete the files from the nodes of Azure dwpool

Question

Hi ,

I have set "RetentionTimeInDays" of azure batch in Azure Data Factory to 0, seeing the answer in this site. But seems like its not working. I can see the files in the Node wd directory even after one day. Is there are any other way to delete the files from node automatically. The reason for deletion is in certain times the node become unusable because of the disk size getting full

Regards
Sijo Johny

Answer 1

Hi @Sijo Johny ,

Thank you for posting query in Microsoft Q&A Platform.

If I understand correctly, you are using custom activity here to run some script on batch pool. Please correct me if I am wrong.

RetentionTimeInDays setting is actually, the retention time for the files submitted for custom activity. Default value is 30 days.

If you are looking of deleting the script files which submitted to batch pool from storage account, then you can consider using Delete activity and configuring dataset to corresponding storage location.

Check below videos to understand delete activity better.
Delete Activity in Azure Data Factory
Delete old files from storage using Azure Data Factory

If you are looking for deleting some files from bath pool nodes then consider writing python or PowerShell script and run that script on bath pool.

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well.