How do I unzip files in a blob from the command in Batch Service?

Anshul Rawat 0 Reputation points
2023-01-23T13:54:41.7666667+00:00

Hello,

I am trying to run a python script from my batch service activity in ADF as follows:

cmd /C python main.py

However, this runs into an error since there are multiple files to be processed by the python script which leads to error code: 2504 [The total size of resourceFiles can't be more than 32,768 characters] ([https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide#error-code-2504)

As a workaround, the documentation suggests to upload a zipped version of all the files and then unzip it in the command as follows:

powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory($zipFile, $folder); }" ;

but this does not work for me and the pipeline gets stuck in processing. I don't understand why this is happening. Is this related to the type of access [admin/nonadmin] of ADF on the batch pool? Either way, please suggest a way of unzipping this file from the batch service activity.

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
301 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,517 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Samy Abdul 3,366 Reputation points
    2023-01-23T14:24:04.7+00:00

    Hi @Anshul Rawat , the limitations of zipping and unzipping files for batch has been discussed below:

    Zipping and UnZipping files for Batch limitation in ADF

    https://learn.microsoft.com/en-us/answers/questions/227787/zipping-and-unzipping-files-for-batch-limitation-i Thanks

  2. AnnuKumari-MSFT 30,601 Reputation points Microsoft Employee
    2023-01-30T04:50:55.0966667+00:00

    Hi Anshul Rawat ,

    Welcome to Microsoft Q&A platform and thanks for posting your question here. As I understand your issue, you are facing an error: "The total size of resourceFiles can't be more than 32,768 characters" , while trying to unzip a file using custom activity. Please let me know if that is not the case.

    Kindly check if the below posts where other customers encountered similar issue has been mitigated , help you resolve the issue:

    Zipping and UnZipping files for Batch limitation in ADF

    ADF Custom Activity : Total size of resourceFiles cannot be more than 32768 characters

    Please let us know if it was helpful. Thankyou.