I have multiple CSV files in adls path, how to compress and decompressed these files with dynamic file names and current timestamp using dataflows in adf

Question

1. I have multiple csv files in adls location
2. How to compress those files with dynamic file names and current timestamp
3. After completing the compressed those files decompressed with dynamic file names and current timestamp again.

Answer 1

Hello, Welcome to MS Q&A

To compress and decompress multiple CSV files in Azure Data Lake Storage (ADLS) with dynamic file names and current timestamps, you can use Azure Data Factory (ADF). Here’s a step-by-step guide:

Compressing Files

Create a Pipeline in ADF:

• Use the Copy Activity to copy the CSV files from the source to the destination.
  • In the Sink settings of the Copy Activity, specify the Compression Type (e.g., GZip or Zip).
    • Use dynamic content to generate the file names with the current timestamp.
    Dynamic File Names with Timestamps

         @concat('compressed_', formatDateTime(utcnow(), 'yyyyMMddHHmmss'), '.zip')
         
  

Decompressing Files

Create Another Pipeline in ADF:

• Use the Copy Activity to copy the compressed files from the source to the destination.
  • In the Source settings of the Copy Activity, specify the Compression Type to indicate that the files should be decompressed.
    • Use dynamic content to generate the decompressed file names with the current timestamp.
    Dynamic File Names with Timestamps References

             @concat('decompressed_', formatDateTime(utcnow(), 'yyyyMMddHHmmss'), '.csv')
             
        
        
    

    References:
• Supported file formats and compression codecs by copy activity in Azure Data Factory and Azure Synapse pipelines
• Data formats supported by Azure Synapse Data Explorer for ingestion (Preview)
• Data formats supported by Azure Data Explorer for ingestion

By following these steps, you can efficiently compress and decompress your CSV files in ADLS with dynamic file names and timestamps using Azure Data Factory.

Please let me know if any further questions

Kindly accept answer if it helps

Thanks

Deepanshu