Azure Data Factory: Copy from Blob to Blob Container with Hierarchical Namespace Fails with 409 Conflict
I am trying to copy data from one storage account to another. I have a pipeline with an activity that copies data from source storage account to sink. The pipeline runs successfully and I can see data has been copied.
But, as soon as I try to copy the data to a storage account with hierarchical namespace enabled the same pipeline and activity does not work.
Please note that I have hierarchical namespace enabled on source storage account in both case. My activity definition looks like below.
{
"source": {
"type": "BinarySource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"deleteFilesAfterCompletion": false
},
"formatSettings": {
"type": "BinaryReadSettings"
}
},
"sink": {
"type": "BinarySink",
"storeSettings": {
"type": "AzureBlobStorageWriteSettings",
"copyBehavior": "PreserveHierarchy"
}
},
"enableStaging": false,
"preserve": [
"Attributes"
]
}
The error response is as below
{
"dataRead": 7622720,
"dataWritten": 0,
"filesRead": 60,
"filesWritten": 0,
"sourcePeakConnections": 64,
"sinkPeakConnections": 11,
"copyDuration": 67,
"throughput": 586.363,
"errors": [
{
"Code": 24116,
"Message": "Failure happened on 'Sink' side. ErrorCode=AzureBlobWriteOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=,Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Blob write operation on path='database' and blobName='db/f1000' failed with error message: The remote server returned an error: (409) Conflict..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (409) Conflict.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=The requested operation is not allowed in the current state of the entity.\nRequestId:57c45a88-401e-007a-66d8-209b12000000\nTime:2024-10-17T21:09:27.2956505Z,,''Type=System.Net.WebException,Message=The remote server returned an error: (409) Conflict.,Source=System,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Blob write operation on path='database' and blobName='db/f1007' failed with error message: The remote server returned an error: (409) Conflict..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (409) Conflict.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=The requested operation is not allowed in the current state of the entity.\nRequestId:71039799-601e-000f-51d8-20f03e000000\nTime:2024-10-17T21:09:27.3130209Z,,''Type=System.Net.WebException,Message=The remote server returned an error: (409) Conflict.,Source=System,'",
"EventType": 0,
"Category": 5,
"Data": {
"FailureInitiator": "Sink"
},
"MsgId": null,
"ExceptionType": null,
"Source": null,
"StackTrace": null,
"InnerEventInfos": []
}
],
"effectiveIntegrationRuntime": "my-integration-runtime (East US 2)",
"usedDataIntegrationUnits": 4,
"billingReference": {
"activityType": "DataMovement",
"billableDuration": [
{
"meterType": "ManagedVNetIR",
"duration": 0.13333333333333333,
"unit": "DIUHours"
}
],
"totalBillableDuration": [
{
"meterType": "AzureIR",
"duration": 0.13333333333333333,
"unit": "DIUHours"
}
]
},
"usedParallelCopies": 32,
"executionDetails": [
{
"source": {
"type": "AzureBlobStorage"
},
"sink": {
"type": "AzureBlobStorage"
},
"status": "Failed",
"start": "10/17/2024, 11:08:20 PM",
"duration": 67,
"usedDataIntegrationUnits": 4,
"usedParallelCopies": 32,
"profile": {
"queue": {
"status": "Completed",
"duration": 52
},
"transfer": {
"status": "Completed",
"duration": 13,
"details": {
"listingSource": {
"type": "AzureBlobStorage",
"workingDuration": 7
},
"readingFromSource": {
"type": "AzureBlobStorage",
"workingDuration": 0
},
"writingToSink": {
"type": "AzureBlobStorage",
"workingDuration": 0
}
}
}
},
"detailedDurations": {
"queuingDuration": 52,
"transferDuration": 13
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
I cannot understand why Iam getting 409, I dont have any other process beside ADF accessing sink bloc storage account. Am I missing some configuration? Thanks in advance.