How to copy individual blob versions from Blob Storage Account to Data Lake account using azure data factory?

Derek Horrall 201

Maybe I am missing it, but how can I copy all blob versions to data lake and not just copy the 'latest' version? Or, how to pull the list of versions, or copy just a specific version? I think i could do a foreach on the list of versions, but I am not sure how to best get the list of versions. API call perhaps? Is there a better way?

MartinJaffer-MSFT 26,061 Reputation points

2022-07-20T16:14:37.897+00:00

Hello @Derek Horrall and welcome to Microsoft Q&A.

As I understand you want to copy all previous versions of a blob in addition to the latest version, while using Azure Data Factory.

Do you want to split each version into its own seperately named blob, or do you want to just take them all as-is?

I know ADLS has RESET APi to see the versions, but I haven't done it with Data Factory before. Also, I think each version has associated metadata, but copying each version might not preserve it. Is that important to you?

2 answers

Derek Horrall 201 Reputation points

2022-07-20T16:27:34.12+00:00

Correct. Imagine I have a blob in blob storage with say, 10 versions over time. I would like to copy those 10 versions to Datalake. So, in the end, my goal would be to have 10 distinct files in Datalake storage.

In my case, I don't care about preserving the meta-data if it is different per each version. In my case, I think they would all be the same anyway.
Please sign in to rate this answer.
MartinJaffer-MSFT 26,061 Reputation points

2022-07-21T21:07:32.843+00:00

Okay @Derek Horrall .

I found how to list the versions using a Web activity, but specifying them in the Copy activity isn't going so well.

The API calls to get the contents of a specific version of a blob it goes like

myAccount.blob.core.windows.net/myContainer/myFile.txt?versionId=2022-07-21T18:57:34.3761209Z

You see the version is specified as url parameter. There isn't exactly anywhere to stick that in Get Metadata or Copy activity or datasets. I tried pasting the "myFile?versionId=2022-07" ... into the file name field and putting it in Get Metadata. I got back "does not exist".

I'm getting a creeping feeling that getting a specific blob version via copy activity might not be possible. There is one more possibility I can think of: abusing the modified datetime filter. However, that is a hail mary pass.

I'll ask around internally, see if anyone else has tried your ask before.

FYI, to get the versions:

myAccount.blob.core.windows.net/myContainer?restype=container&comp=list&prefix=myFile.txt&include=versions&INSERT_SAS_TOKEN_HERE
Sign in to comment
MartinJaffer-MSFT 26,061 Reputation points

2022-07-21T21:23:30.05+00:00

I thought of another solution. This task of 'flattening' versions is probably a one-off thing. Why not just make use of the blob REST api's copy command? copy-blob

Or even AzCopy
Please sign in to rate this answer.
Derek Horrall 201 Reputation points

2022-08-04T19:44:22.433+00:00

So, you mean

Use this to get the versions: myAccount.blob.core.windows.net/myContainer?restype=container&comp=list&prefix=myFile.txt&include=versions&INSERT_SAS_TOKEN_HERE

Then iterate through the versions and call copy-blob somehow? Could that be with a for-each?

MartinJaffer-MSFT 26,061 Reputation points

2022-10-05T16:50:36.563+00:00

Yes that would be a for-each web activity. @Derek Horrall
Sign in to comment

Share via

How to copy individual blob versions from Blob Storage Account to Data Lake account using azure data factory?

2 answers