Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
Message Exceeded Limit Azure Synapse Pipeline
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 13%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Malte Thesenvitz
1
Reputation point
I'm running a GET request to an API via an Synapse pipeline and a copy activity (Source: REST API, Sink: Parquet file). The vast majority of the time the pipeline and the copy activity run fine, but sometimes I'm getting the following error message:
Failure happened on 'Source' side. ErrorCode=RestResourceReadFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Fail to read from REST resource.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=The underlying connection was closed: The message length limit was exceeded.,Source=System
I've tried manually sending the API request that's causing the above error message and I am getting a successful response from the system. Additionally, when I run the same API request via the Azure pipeline a second time, it runs fine.
Thanks in advance for any guidance on this.
Azure Synapse Analytics
{count} votes
-
ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator2022-12-15T08:12:34.497+00:00 Hi @Malte Thesenvitz ,
Thank you for posting query in Microsoft Q&A Platform.
Above error may come, when your working with large data. In this particular case, is your API response or Request is huge? If yes, Is it possible to play with your API to make multiple API calls which helps to send large data in small chunks?
-
Malte Thesenvitz 1 Reputation point
2022-12-16T10:44:22.59+00:00 Hi Shaik, thanks for getting back.
We've already separated our calls on an hourly basis (meaning we only pull one hour worth of data each time, averaging file sizes less than 1MB) and we are still receiving the same error message from above..
-
Malte Thesenvitz 1 Reputation point
2022-12-19T09:27:51.623+00:00 Hi Shaik,
we found the issue for the failure. In our response we are being flooded with status headers from the API. This makes our message exceed the limit, hence the error message. I've attached a screenshot below.
Is there a way to disable the response header returned via the API call in Azure's copy activity?
-
ShaikMaheer-MSFT 38,556 Reputation points Microsoft Employee Moderator
2022-12-22T09:49:05.727+00:00 ADF is acts as client tool here to make API call and get response back. So unfortunately, thats not possible from ADF side, until API has some facility to pass some data in request and accordingly gives you back response. Kindly check with API team side if thats possible. Thank you.
Sign in to comment
Sign in to answer