Datafactory pipeline copying only 2k rows to snowflake sink

Question

Hello folks,

I'm working on a project to integrate xml to snowflake using datafactory

context: i have a pipeline who loop over all the xml files and copy it to snowflake the xml file can be 20koctet to 1 giga of data.

the xml are located on a adlsgen2 and i'm able to copy rows to the snowflake table.

But there is a problem when i check the copy activity of the snowflake sink it say: data written 1mb and 2 k rows written while the xml file is 33mb.

and this is not finish when i check the over copy activity this is the same number everywhere when is check the snowflake rows written it is always 2 k rows and 1 mb.

Can someone help me to understand what happend and how to resolve the issue ?

bellow the image

Answer 1

Hi Sacha CLEMENTE:

Thanks for reaching out to Microsoft Q&A.

when i check the copy activity of the snowflake sink it say: data written 1mb and 2 k rows written while the xml file is 33mb.

it necessarily need not have to match the exact size of the xml file, the original content that is available inside the xml file to be written row-by-row may be only 2k rows which might constitute to only ~2mb.

Pls note that ADF by design does not have any restrictions on the amount of data that can be transferred in the pipelines. Also note that it succeeded and you dont have a failure of this pipeline hence it proves that the entire ingestion was complete with readable rows that can be inserted into the SF table.

Please 'Upvote’(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.