How to copy individual blob versions from Blob Storage Account to Data Lake account using azure data factory?

Derek Horrall 196 Reputation points
2022-07-18T15:10:19.487+00:00

Maybe I am missing it, but how can I copy all blob versions to data lake and not just copy the 'latest' version? Or, how to pull the list of versions, or copy just a specific version? I think i could do a foreach on the list of versions, but I am not sure how to best get the list of versions. API call perhaps? Is there a better way?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,072 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Derek Horrall 196 Reputation points
    2022-07-20T16:27:34.12+00:00

    Correct. Imagine I have a blob in blob storage with say, 10 versions over time. I would like to copy those 10 versions to Datalake. So, in the end, my goal would be to have 10 distinct files in Datalake storage.

    In my case, I don't care about preserving the meta-data if it is different per each version. In my case, I think they would all be the same anyway.


  2. MartinJaffer-MSFT 25,991 Reputation points
    2022-07-21T21:23:30.05+00:00

    I thought of another solution. This task of 'flattening' versions is probably a one-off thing. Why not just make use of the blob REST api's copy command? copy-blob

    Or even AzCopy