Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Azure DataFactory: Error while copying data from http source to ADLS Gen2
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 85%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anisha Singhal
1
Reputation point Microsoft Employee
I am trying to copy the data from an http source to ADLS gen2 and I get below error:
Error: Failure happened on 'Sink' side. ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server：The remote server returned an error: (403) Forbidden.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (403) Forbidden.,Source=System,'
There is a foreach loop within which the copy from http to ADLS gen2 takes place. This loop works for few records but fails after some time. The configuration settings for the copy data task is as shown below:
Please let me know if there is a fix for this?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT 26,086 Reputation points2021-04-23T18:34:59.717+00:00 Hello @Anisha Singhal and welcome to Microsoft Q&A.
To help narrow down the cause, I have a few questions.
- Is it always the same page that fails?
- You are copying inside a loop. Does the source change with each iteration, or are you copying the exact same source data each time?
- Is the loop sequential, or in parallel?
- How much time passes before it starts to fail?
- Is there any authentication involved with your http source?
- The page that fails, can you read it with something other than ADF? (Such as Postman or Curl)
-
MartinJaffer-MSFT 26,086 Reputation points
2021-04-27T19:37:07.623+00:00 @Anisha Singhal I haven't heard back from you. Are you still facing the issue?
-
MartinJaffer-MSFT 26,086 Reputation points
2021-04-28T16:48:45.917+00:00 @Anisha Singhal if you found your own solution, please share here with the community.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 3%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Chethan 6 Reputation points2021-07-27T20:36:58.227+00:00 I am facing the same issue while copying data from Gen1 to Gen2 using ADF pipeline. Copy process starts and after copying few files it fails with Forbidden erroron sink side on a file. When u look for the file on Sink (Gen2) the file is copied too.
Not sure what is causing the failure.Error:
Failure happened on 'Sink' side. ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'Forbidden'. Account: 'xxxx'. FileSystem: 'xx'. Path: 'xxx'. ErrorCode: 'AuthorizationPermissionMismatch'. Message: 'This request is not authorized to perform this operation using this permission.'. RequestId: '69301b22-901f-0009-5125-837848000000'. TimeStamp: 'Tue, 27 Jul 2021 20:26:25 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Operation returned an invalid status code 'Forbidden',Source=,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'Forbidden',Source=Microsoft.DataTransfer.ClientLibrary,'
-
MartinJaffer-MSFT 26,086 Reputation points
2021-07-28T18:29:05.627+00:00 Hello @Chethan and welcome to Microsoft Q&A.
Could you please open a new question thread, and we will help you there.
This thread is old, and your error message seems slightly different.
Thank you.
Sign in to comment
Sign in to answer