Welcome to Microsoft Q&A platform and thanks for posting your query here.
As I understand you're facing an issue with the Lookup component in your ADF pipeline where it’s throwing an error due to exceeding the data size limit, and as a result, the subsequent email component isn't being triggered.
To assist you further, could you please let me know the file size? Are you sure it doesn’t contain more than 5,000 rows? If it does, consider splitting the file into smaller files (under 4 MB each), though this may not be practical for a large number of files.
Just to clarify:
- The Lookup activity can return up to 5,000 rows. If the result contains more records, only the first 5,000 rows will be returned.
- The output from the Lookup activity can be up to 4 MB in size; it will fail if it exceeds this limit.
- The maximum duration for the Lookup activity before timing out is 24 hours.
You can check the details here: Lookup Activity Capabilities.
You might also want to try the workaround mentioned in this link: Stack Overflow Workaround.
Watch out this video for reference: Handle Lookup with more than 5000 records using Loop
I hope this helps. Please let me know if you have any further questions or concerns.