Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)

Bhuvnesh Kumar 26 Reputation points
2020-09-20T16:47:33.427+00:00

I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated time for all files as GEN1 lake.

I was looking to use ADF for this use case. But for that we need to define dataset, and to define dataset we have to define data format(Avro,json,xml, binary etc). As we have different type of data mixed, I tried to use binary format. But with binary format all file at destination have content type "application/octate-stream". Also not able to retain file update time.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,337 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,508 questions
{count} votes

Accepted answer
  1. HarithaMaddi-MSFT 10,126 Reputation points
    2020-09-29T07:01:03.41+00:00

    Hi @Bhuvnesh Kumar ,

    Thanks for your patience. Apologies for the confusion in above responses.

    I got confirmation from Product team that Last Modified Time is system metadata that represents that modification in the filesystem/container and it cannot be updated. Adding user meta data to capture meta data from the source is work around and all the above references will work for adding and updating additional property. Please find below the workaround implemented in PowerShell, kindly suggest for any issues in implementing this in your requirement.

    28968-powershell.png

    Hope this helps!


0 additional answers

Sort by: Most helpful