Getting this error "The size 4194822 of lookup activity result exceeds the limitation 4194304" while processing the CSV file to get the RowCount

ajit more (PSP) 40 Reputation points
2023-10-10T08:12:19.1333333+00:00

Getting this error:

ErrorCode=UserErrorComputePluginFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The size of lookup activity result exceeds the limitation 4194304.,Source=,''Type=Microsoft.DataTransfer.Execution.Core.ExecutionException,Message=The size of lookup activity result exceeds the limitation 4194304.,Source=Microsoft.DataTransfer.TransferTask,'

User's image

User's image

But actually, the CSV file has only 4k rows.

Sample file:- AM-sample.txt

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,711 questions
0 comments No comments
{count} votes

Accepted answer
  1. Vinodh247-1375 11,396 Reputation points
    2023-10-10T09:41:02.72+00:00

    Hi ajit more (PSP):

    Thanks for reaching out to Microsoft Q&A.

    What is the fize size? are you sure it doesn't has more than 5k rows? if yes you can try to break the file into smaller multiple small size files (under 4mb) and try but wont be feasible solution for huge amount of files.

    • The Lookup activity can return up to 5000 rows; if the result set contains more records, the first 5000 rows will be returned.
    • The Lookup activity output supports up to 4 MB in size, activity will fail if the size exceeds the limit.
    • The longest duration for Lookup activity before timeout is 24 hours.

    https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity#supported-capabilities

    Also you can give a try the mentioned workaround in the link below.

    https://stackoverflow.com/a/55856185/10653466

    Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

    1 person found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful