Scripts activity is processing only 5000 records but source table is having 20K+ Records

Question

Hi @Nandan Hegde and Experts,

We have a requirement to read on-premise oracle data and send this output to perl team in JSON format.

For this we used Script activity and inside that the query logic is added and query is able to return results but problem is it is returning only 5000 records.

While i go through some blogs people are talking about lookup activity is having some limitations hence it allows only 5k records but in our pipelines except one pipeline we no where used lookup instead it is pretty straight forward only script activity is used but not sure why only 5000 records are pulling whereas it's source tables is having more than 25k records and it is valid that production environment tables might have millions of records also.

Could anyone please advise how to overcome this 5000 records process scenario? I need to process all the records from source irrespective of its count.

Thanks in advance for your help.

Regards,
Kumar

Answer 1

Hello! Thank you for your question! This question was already discussed here

https://learn.microsoft.com/en-us/answers/questions/42083/how-to-process-more-than-5000-records-in-lookup-ac.html

Answer 2

Hi @Kumar

This video explains the steps to handle the situation.

Hope it helps.

Thanks