Error from Lookup component in ADF not invoking subsequent component

Huckleby, Barbara A 0 Reputation points
2024-10-29T16:08:06.9533333+00:00

We have a component running a stored procedure. This particular one returned too much data, which is throwing this error:

ErrorCode=UserErrorComputePluginFailure,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The size of lookup activity result exceeds the limitation 4194304.,Source=,''Type=Microsoft.DataTransfer.Execution.Core.ExecutionException,Message=The size of lookup activity result exceeds the limitation 4194304.,Source=Microsoft.DataTransfer.TransferTask,'

The Error port is attached to a following component, which will generate an email letting us know something broke. However, that component is not being executed. All of the prior components also have their failed port attached to the same email component.

We can't figure out why this pipeline is not invoking that 'email' component.

The same thing happens if one of the components running a stored procedure times out.

Any assistance would be appreciated.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,811 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Smaran Thoomu 16,640 Reputation points Microsoft Vendor
    2024-10-29T17:43:13.07+00:00

    Hi @Huckleby, Barbara A

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    As I understand you're facing an issue with the Lookup component in your ADF pipeline where it’s throwing an error due to exceeding the data size limit, and as a result, the subsequent email component isn't being triggered.

    To assist you further, could you please let me know the file size? Are you sure it doesn’t contain more than 5,000 rows? If it does, consider splitting the file into smaller files (under 4 MB each), though this may not be practical for a large number of files.

    Just to clarify:

    1. The Lookup activity can return up to 5,000 rows. If the result contains more records, only the first 5,000 rows will be returned.
    2. The output from the Lookup activity can be up to 4 MB in size; it will fail if it exceeds this limit.
    3. The maximum duration for the Lookup activity before timing out is 24 hours.

    You can check the details here: Lookup Activity Capabilities.

    You might also want to try the workaround mentioned in this link: Stack Overflow Workaround.

    Watch out this video for reference: Handle Lookup with more than 5000 records using Loop

    I hope this helps. Please let me know if you have any further questions or concerns.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.