how to delete a folder in ADLS with different name

17285674 0 Reputation points
2023-03-13T14:15:03.5433333+00:00

Hi,

I have a pipeline who generate parquet file in ADLS gen2 with similar naming patternfor each folder every day. bust some times a bug cause an empty folder generate in all folders with different naming. I would like to use delete activity befor my pipeline to delete folder with different naming pattern.

can you help me in this please?

thank you

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,330 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,316 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,901 questions
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee
    2023-03-14T09:07:28.2466667+00:00

    Hi @17285674 ,

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    As I understand your issue, you want to delete the folders which are empty and have some random names.

    You can try the following approach to achive that:

    1. Use Get metadata activity where dataset is not pointing to any container or any folder. Select 'Child Items' in the field list to fetch the list of all the containers in the ADLS account. User's image
    2. Use ForEach activity to iterate through the output of get metadata activity using the expression :
       @activity('Get Metadata1').output.childItems
    
    1. Use Another GetMetadata activity inside ForEach and make the dataset parameterized to get all the file names within each of the folders. User's image
    2. Use If Block to check if the step3 output is empty , which means if the folder doesn't contain any file, then use delete activity inside If block to delete that particular folder. Use the following expression in If condition :
       @empty(activity('Get Metadata2').output.childItems)
       
    

    User's image


    Hope it helps. Kindly accept the answer and mark it as helpful.

    1 person found this answer helpful.