Data Factory Copy data Blob - To _Table Storage

Question

Hi Team,

I am running a data factory job that loads data from Blob Storage to Table storage, I am using 200 Data integration Unit.

I am getting the below error.

Operation on target Copy_8fe failed: Failure happened on 'Sink' side. ErrorCode=FailedStorageOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A storage operation failed with the following error 'Element 0 in the batch returned an unexpected response code.'.,Source=,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=Element 0 in the batch returned an unexpected response code.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=The server is busy.
RequestId:172be358-f002-0057-1ada-273ff6000000
Time:2021-04-02T16:05:34.6782152Z,,'

Please help me resolve this issue

Answer 1

Hello again @Imran Mondal .

It sounds like you are being throttled by Table Storage. 200 Data integration units sound like enough parallelism to cause this, assuming each unit moves 100 entities per second, or all of them move 10 entities per second into the same partition. See table storage performance limits here.

Maximum request rate per storage account 20,000 transactions per second, which assumes a 1-KiB entity size
Target throughput for a single table partition (1 KiB-entities) Up to 2,000 entities per second

In your case, I think reducing the Data integration units might actually be the solution. If you still need more throughput, options include using more storage accounts, bigger storage accounts, or changing architecture.