Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
How to specify numbers of recors,rows to read in a copy activity?
Pablo CR
21
Reputation points
How can I set the number of records to process so my Copy Activity won't crash.
I'm copying data from Dynamics365 to Azure BLOB but i think the data in Dynamics is so big so Azure datafactory can't read it and it doesn't copy nothing
here more info as you can see there is nothing read yet and time has passed by 12 minutes
I think, again it's because data is too big.
I need to copy all data(Table) in Origin(by the way it doesn't matter if the Sorce to place the data is a BLOB or a other SQL). Eventually the activity trows an error.
Should I make n "Activities copy" and then instead of table selected in "Use Query" use a query to narrow the data to be copy?
how to make like chunks/buckets of data to be sent but with all the table?
{count} votes
-
HimanshuSinha-msft 18,366 Reputation points Microsoft Employee
2021-07-02T22:54:34.943+00:00 Hello @Pablo CR ,
Thanks for the ask and using the Microsoft Q&A platform .There are few things which I am not sure here . You are getting an ERROR code of 502 bad gateway , if the error could be due to huge data i was expecting to have a timeout error . Anyways you are on the right path , lets do a quick test and use query and try to pull a table/entity whcih is small . If the smaller recordset also fails then may be we are looking at the right place .
Please do let me know how it goes .
Thanks
Himanshu
Sign in to comment
But how can I tell to Azure data factory to read packages like 100 to 100 a so on? That integrations is conected to Dynamics365 Online fetching data by default it gives 5000 records but as the data is to heavy seems like the service in dynamics crash i dont know why, but with small chunks it works well, there is something called paging in dynamics FetchXml that allow to bring data in pages so this could make the result more lighter and it does. but ADF don't allow paging
the amount of records I'm trying to move are 10,000+ so building n copy activities like 500 per activity would be 10,000/500 = 20 activities! if the records were 1,000,000 should I make 200 activities?
Hello @Pablo CR ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.
Thanks
Himanshu