How can i append files from azure blob storage to a file present in file share using azure data factory ?

Question

I want to use Azure data factory to append multiple files present in a blob container to a file present in file share. As the size of the file is big i want to go with Azure data factory .I am creating a data set of file system and have chosen binary format of data . I don't find any option to append into file share.

Answer 1

Hello @Pratik Agrawalla and welcome to Microsoft Q&A.

I'm sorry to say, your ask is not possible. Please allow me to explain why.

Correctly appending data is not simply smushing one BLOB against another BLOB. Especially so with binary, as binary can be any file type, any content type, any encoding. Binary moves as-is without parsing or interpreting or any processsing. If binary could append, it would open the door for nonsensical combinations like appending sound to an image or executable.

Even for two of the same file type, it is more complicated.
Parquet has special sections that could be invalidated by the append.
Delimited text can have header rows, and a simple smush would place the second header in the middle of the resulting file, breaking the ability for downstream applications to ingest it.

Currently you cannot append data in blob storage.

The next best thing you could do, is place the 2 files next to each other, then do another copy activity using both of them together as source. In the settings you can merge the files.

The difference between merge and append, is that everything to combine is in source for merge. Append expects adding to, not overwriting the sink target. You would need to use something other than binary for this.