ADF CopyActivity for tar file Http Source to Blob Sink

Filmon Belay 0 Reputation points Microsoft Employee
2023-04-21T12:47:32.21+00:00

Hi, I am using the ADF c# SDK to try and copy a single 15GB tar file from an http address to blob storage. I am using an Http and Azure Blob Storage connector for source & sink and Binary Datasets for both connectors. However, when I look at the file in Blob Storage, I see the file size repeatedly go up to ~2 MB then reset to 1015B, eventually the size of the file does not increase in Blob storage and stays at 1015B, even though the monitor pipeline shows that more bytes are being written.

Have I misconfigured something somewhere? I've attached the configuration for the pipeline/datasets/linked services in the question.

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,497 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,778 questions
{count} votes