Scripts activity is processing only 5000 records but source table is having 20K+ Records

Kumar 41 Reputation points
2022-10-19T11:45:01.047+00:00

Hi @Nandan Hegde and Experts,

We have a requirement to read on-premise oracle data and send this output to perl team in JSON format.

For this we used Script activity and inside that the query logic is added and query is able to return results but problem is it is returning only 5000 records.

While i go through some blogs people are talking about lookup activity is having some limitations hence it allows only 5k records but in our pipelines except one pipeline we no where used lookup instead it is pretty straight forward only script activity is used but not sure why only 5000 records are pulling whereas it's source tables is having more than 25k records and it is valid that production environment tables might have millions of records also.

Could anyone please advise how to overcome this 5000 records process scenario? I need to process all the records from source irrespective of its count.

Thanks in advance for your help.

Regards,
Kumar

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,015 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,893 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Yuliya Chernomordikova 1 Reputation point
    2022-10-19T12:38:35.493+00:00

    Hello! Thank you for your question! This question was already discussed here

    https://learn.microsoft.com/en-us/answers/questions/42083/how-to-process-more-than-5000-records-in-lookup-ac.html

    0 comments No comments

  2. Suba Balaji 11,196 Reputation points
    2022-10-19T13:00:43.583+00:00

    Hi @Kumar

    This video explains the steps to handle the situation.

    Hope it helps.

    Thanks

    0 comments No comments