Retaining all files attributes like ACLs, File Owner & etc in Blob Storage

LoganIN 61

Hi,

We are using Azure Data Factory to copy the objects from local server to Azure Blob storage. We were able to copy the objects successfully using the copy-activity in ADF, but the file attributes like ACLs, TimeStamp, File Owners are missing . I couldn't see any article which says the file attributes retention, Can someone please confirm the same and if yes, which tool we can use to preserve the same.

Thank you in advance!

Regards,
Logan

2 answers

MartinJaffer-MSFT 26,021 Reputation points

2022-01-28T22:25:48.837+00:00
Hello @LoganIN and welcome to Microsoft Q&A.

There is an option for preserving the metadata you mentioned. See Here. This option applies only to binary dataset because other types allow mapping, which can change contents, and make the metadata invalid.

When you configure source as Data Lake Storage Gen1/Gen2 with binary format or the binary copy option, and sink as Data Lake Storage Gen2 with binary format or the binary copy option, you can find the Preserve option on the Settings page in Copy Data Tool or on the Copy Activity > Settings tab for activity authoring.

Thanks, Martin

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Please sign in to rate this answer.
LoganIN 61 Reputation points

2022-01-29T11:36:35.52+00:00

Hi @MartinJaffer-MSFT ,

Thank you for your response.

I have gone through the link and it seems applicable only for the below mentioned items, not for the Local file system like FTP or SFTP Servers as source.

When you copy files as-is from Amazon S3/Azure Data Lake Storage Gen2/Azure Blob storage/Azure Files to Azure Data Lake Storage Gen2/Azure Blob storage/Azure Files with binary format, you can find the Preserve option.

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-preserve-metadata#preserve-metadata

My source location is the SFTP and we would like to archive those files, at the same time all of the attributes like Time Stamp, ACLs & Owner & etc should be retained in case if there is any restoration.

Thanks in Advance!

Regards,
Logan

MartinJaffer-MSFT 26,021 Reputation points

2022-01-31T16:37:45.73+00:00

Hmm okay. I'll take a look at that, but I do have some off-the-cuff comments, @LoganIN .

Namely, what should happen if the owner's identity exists in the SFTP source, but not in the sink data store? If the owner might not be valid, then the ACL's also would become invalid. Those two features likely will not transfer over. The reason it works between Azure storage services, is that Azure Active Directory helps with the identities (this statement needs verification).

The Time Stamp has a much better chance of preservation. Do specify what this Time Stamp is describing (creation or last update / modified on). Last update / modified on is a much more widespread feature than creation date.

LoganIN 61 Reputation points

2022-02-01T17:32:33.03+00:00

Thanks @MartinJaffer-MSFT , Awaiting your reply for such a scenario.

If it doesn't copy the ACLs then we can't migrate the data as it's the primary requirement. Now the timestamp we are looking for the last modified only but i don't see any option in ADF to copy those attributes.

MartinJaffer-MSFT 26,021 Reputation points

2022-02-02T01:46:53.62+00:00

Ouch. Things aren't looking so good @LoganIN .

This difficulty of transferring ACL's isn't just an Azure thing. Even if you were going from one SFTP to another SFTP, you would still run into the same issue. Unless the identity provider is shared, you will need to re-construct all the users in the destination. It is like if you wanted to move your friend list between chat clients, like between Slack and Skype. If you friend on Slack doesn't have Skype, then Skype doesn't know who to send messages to.

So, there isn't any easy button I know of for this, but I can imagine a process to see your migration through. Here is a high-level view:

Make each account from the SFTP have an identity in Azure.

Create a spreadsheet / SQL table / CSV document mapping the related identities.

Do the same with groups.

Use a script to copy the files and set the ACL's / permissions, leveraging the mapping to translate the identities as appropriate.

I know it is messy. @Saurabh Sharma do you know of any identity-migration tools?

Loganathan V 1 Reputation point

2022-02-23T12:36:11.713+00:00

I know it's quite old but i just wanted to conclude this question.

I can understand the ACL's part but what about the attributes Like File Owner, File Modified date, file created date are not copied to Azure.
Sign in to comment
MartinJaffer-MSFT 26,021 Reputation points

2022-02-28T17:24:53.187+00:00

File Owner falls under the same issue as with ACL.

Created date is tied to the file system, at least on a low level. At the most base level, created date is not "When was this document created?" , but instead "When was this section of disk memory allocated to hold data?". The difference between these two questions, is the difference between talking about the sheet of paper a document is written on versus talking about the content of the document. When copying data from one system to another, you are creating a new document with contents identical to the old one. This the created date in the destination is when the copy was made.

I did chat with my colleague, and he suggested using Azure Files and AzCopy rather than Azure Blob and Data Factory, as this may have the capabilities you want. See discussion thread. @Loganathan V
Please sign in to rate this answer.

0 comments No comments
Sign in to comment