Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)

asked 2020-09-20T16:47:33.427+00:00
Bhuvnesh Kumar 26 Reputation points

I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated time for all files as GEN1 lake.

I was looking to use ADF for this use case. But for that we need to define dataset, and to define dataset we have to define data format(Avro,json,xml, binary etc). As we have different type of data mixed, I tried to use binary format. But with binary format all file at destination have content type "application/octate-stream". Also not able to retain file update time.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
894 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,266 questions
{count} votes

Accepted answer
  1. answered 2020-09-29T07:01:03.41+00:00
    HarithaMaddi-MSFT 10,071 Reputation points

    Hi @Bhuvnesh Kumar ,

    Thanks for your patience. Apologies for the confusion in above responses.

    I got confirmation from Product team that Last Modified Time is system metadata that represents that modification in the filesystem/container and it cannot be updated. Adding user meta data to capture meta data from the source is work around and all the above references will work for adding and updating additional property. Please find below the workaround implemented in PowerShell, kindly suggest for any issues in implementing this in your requirement.

    28968-powershell.png

    Hope this helps!


0 additional answers

Sort by: Most helpful