Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)

Bhuvnesh Kumar 26

I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated time for all files as GEN1 lake.

I was looking to use ADF for this use case. But for that we need to define dataset, and to define dataset we have to define data format(Avro,json,xml, binary etc). As we have different type of data mixed, I tried to use binary format. But with binary format all file at destination have content type "application/octate-stream". Also not able to retain file update time.

HarithaMaddi-MSFT 10,126 Reputation points

2020-09-21T12:57:21.747+00:00

Hi @Bhuvnesh Kumar ,

Welcome to Microsoft Q&A Platform.

Please follow the instructions in this MS documentation that explains creating a pipeline using a template. Using this approach, I was able to copy .csv,.xlsx,.json files from gen1 to gen2 using a single activity. And to retain the last modified time, there is no inbuilt ability today in copy activity to do that. Using "Get Metadata" activity, the last modified date can be retrieved for all the files and calling the REST API to add a property is one approach to achieve the same. I would recommend you to upvote this idea. Data Factory team will look at them to implement in future releases. Please let us know for further queries ,we will be glad to assist.
Bhuvnesh Kumar 26 Reputation points

2020-09-21T14:03:30.647+00:00

Hey What file format you used for copying multiple file types?
Also I can see there is "Meta Data Fetch" activity in ADF, But how we can use Rest Api in same pipeline ? How you are suggesting to use Rest-API, you mean for all file sun some script to update date?
HarithaMaddi-MSFT 10,126 Reputation points

2020-09-21T14:11:46.027+00:00

Hi @Bhuvnesh Kumar ,

I have used delimited text format and it worked for copying those files without any issue, please find attached the pipeline JSON. The idea to update the last modified date is by using "Get Metadata", "Foreach" and "Web Activity" which can be used to update this for every file. These can be used in the single pipeline along with copy activity. This is the documentation for REST API.

Hope this helps! Please let us know for further queries and we will be glad to assist.

pipelinejson.txt
Bhuvnesh Kumar 26 Reputation points

2020-09-22T06:01:36.503+00:00

Hi @HarithaMaddi-MSFT

There is no such header to set lastModifiedDate in setBlobProperty.
Do you mean "x-ms-meta-name:value" header in setBlobMetaData?
HarithaMaddi-MSFT 10,126 Reputation points

2020-09-22T09:10:23.037+00:00

Hi @Bhuvnesh Kumar ,

Thanks for sharing the query. I found better ways of doing this using PowerShell and .Net commands as below articles. Please suggest if these helps in implementing the requirement.

storage-blob-container-properties-metadata Ref: how-to-update-file-last-modified-date-in-azure-blob-storage
set-metadata-on-azure-blob-storage-containers-powershell
Bhuvnesh Kumar 26 Reputation points

2020-09-23T05:13:02.737+00:00

yes, same functionality also available in java sdk.
https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-java-sdk
HarithaMaddi-MSFT 10,126 Reputation points

2020-09-23T16:53:39.81+00:00

Thanks @Bhuvnesh Kumar . Please let us know if above details are helpful, if not, we will continue to engage with you on the question.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
HarithaMaddi-MSFT 10,126 Reputation points

2020-09-25T09:11:33.08+00:00

Hi @Bhuvnesh Kumar ,

We have not received a response from you. Please let us know if above response is helpful, if not, we will continue to engage with you on the question.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Bhuvnesh Kumar 26 Reputation points

2020-09-28T12:13:12.377+00:00

Not worked.

I was trying to use other alternative but not worked (https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-metadata)

curl -XPUT 'https://<accountName>.dfs.core.windows.net/<FileSystem>/<filePath>?comp=metadata' -H'Authorization: Bearer <Token from first Request>' -H'Date: Thu, 24 Sep 2020 09:49:37 GMT' \
-H'x-ms-version: 2018-11-09' \
-H'x-ms-meta-category: test' \
-d'{}'
Response - {"error":{"code":"MissingRequiredHeader","message":"An HTTP header that's mandatory for this request is not specified.\nRequestId:57bf1e7a-201f-006f-6d30-927d37000000\nTime:2020-09-24T05:05:49.6547386Z"}}
Bhuvnesh Kumar 26 Reputation points

2020-09-28T12:20:14.857+00:00

One more API I tried -
(https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update)

curl -XPATCH 'https://<AccountName>.dfs.core.windows.net/<FileSystem>/<FilePath>?action=setProperties' -H'Authorization: Bearer <TOken>' -H'Content-type: application/json' -H'x-ms-properties: lastmodificationtime=1601273655833'
{"error":{"code":"InvalidHeaderValue","message":"The value for one of the HTTP headers is not in the correct format.\nRequestId:7a146177-d01f-0044-3b7a-95c913000000\nTime:2020-09-28T09:33:12.2037773Z"}}
Bhuvnesh Kumar 26 Reputation points

2020-09-28T12:20:26.95+00:00

Tried many combination for this header, not sure what is error.

I can also partially resolve issue if I can migrate LMT from AWS to GEN1 lake, for that I tried JAVA sdk's setTime function, that is return success response but New date not reflecting on file -
(https://azure.github.io/azure-data-lake-store-java/javadoc/com/microsoft/azure/datalake/store/ADLStoreClient.html#setTimes-java.lang.String-java.util.Date-java.util.Date-)
HarithaMaddi-MSFT 10,126 Reputation points

2020-09-28T13:50:56.64+00:00

Hi @Bhuvnesh Kumar ,

Thanks for sharing the details. Apologies for the inconvenience.

This is to update that I am working on it and also discussing with Product team and will get back to you if I hear any alternative on the same.

Thanks for your patience!

Accepted answer

HarithaMaddi-MSFT 10,126 Reputation points

2020-09-29T07:01:03.41+00:00

Hi @Bhuvnesh Kumar ,

Thanks for your patience. Apologies for the confusion in above responses.

I got confirmation from Product team that Last Modified Time is system metadata that represents that modification in the filesystem/container and it cannot be updated. Adding user meta data to capture meta data from the source is work around and all the above references will work for adding and updating additional property. Please find below the workaround implemented in PowerShell, kindly suggest for any issues in implementing this in your requirement.

Hope this helps!
Please sign in to rate this answer.
Bhuvnesh Kumar 26 Reputation points

2020-09-29T07:10:41.99+00:00

Thanks @HarithaMaddi-MSFT

As now it's confirmed that it's limitation on lake, I will use this user-metaData work-around.
Sign in to comment

Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)

0 additional answers