How to rename file name in bulk while copying it from gen1 data lake to gen2 data lake.

Kavita Kumari 5 Reputation points
2023-05-08T14:34:19.04+00:00

Requirement : Copy all the files starting with name "DATA_<timestamp>.parquet " available in gen1 path to gen2 path , while copying the files we have to rename the files . Replace "DATA<timestamp>.parquet" with "GEN2-DATA-FILE__<timestamp>.parquet" . Timestamp should be the old one only.

Steps I followed :

  1. fetch all metadata available in gen1 path
  2. Filter with filename using @statswith() function
  3. applied for each and used copy activity
  4. configured source and sink in copy activity
  5. facing issue in sink I tried using replace() function for filename but its not giving any result. For each activity getting executed but I am not getting any output

filename expression I used : @replace(item.name,'DATA','GEN2-DATA-FILE')

Please let me know what expression I should use to achieve the result

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2023-05-09T05:06:30.4966667+00:00

    Hello @Kavita Kumari ,

    Thanks for the question and using MS Q&A platform.

    I was able to get this done with parametrized dataset.

    User's image

    The below screenshot will help you understand the implementation .

    RanemFile

    Himanshu

    Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues. 

    1 person found this answer helpful.