ADF MongoAtlas Source side error when batch size existed

Question

ADF MongoAtlas Source side error when batch size existed

Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM) 0

I have to copy data from MongoDB source side but I got an error as below.

I can run this pipeline if I remove the batchSize. But the batch size gets added automactically after I save the pipeline and the error shows up again when I re-run the pipeline.

User's image

0 comments

1 answer

Your answer

Answer 1

ShaikMaheer-MSFT 38,631 Microsoft Employee Moderator

Hi Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM),

Thank you for posting query in Microsoft Q&A Platform.

Invalid cast error usually occurs when there is a mismatch between data type of source and destination columns.

To resolve this issue, you can try following:

Ensure datatype of source and sink columns matches. Leverage Mapping tab in Copy activity.
Check if there are any null values in source column. If that causing any issue because of sink column may be not null then try to using Fault Tolerance in Settings tab in copy activity to skip incompatible rows.

Hope this helps. Please let me know how it goes.

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM) 0 Reputation points

2023-09-05T09:31:22.85+00:00

Hi @ShaikMaheer-MSFT ,

Thanks for your answer. But I haven't got any problem with mapping info. If I remove batch size from source tab, the pipeline works normally.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-09-06T08:34:18.9233333+00:00
Hi Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM),
Looks strange. As per documentation, default batch size is 100. That means even if you remove batch size and save it product may consider batch size of 100. click here for documentation.

What is the batch size you are trying?

After you remove batch size in debug run its working fine? if yes, kindly check input json of copy activity after execution and see if any batch size mentioned there?

After you save or publish pipeline, you are saying batch size is again getting added. Could you please share what is the value its getting added there?
Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM) 0 Reputation points

2023-09-07T03:46:58.53+00:00

Hi @ShaikMaheer-MSFT ,

I just use the default batch size.

Here is the input when I run this pipeline with batch size:

And without batch size:

All other things are the same.

The pipeline was working fine before. I have just updated the dataset to connect to source via IR instead of public network for security reasons. And now the error occurs as I detailed above.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-09-07T17:04:18.47+00:00

Hi Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM),

Thank you for sharing details. Could you please try with keeping batch size explicitly as 1 or 0 and see if that helps?
Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM) 0 Reputation points

2023-09-08T02:44:09.7533333+00:00

I have tried 0, 1, 1000, -1. The pipeline is still error.
Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM) 0 Reputation points

2023-09-11T02:53:44.1+00:00

Hi @ShaikMaheer-MSFT ,

Please kindly check this error. I have to spend a lot of effort to run these pipelines manually every day. Let's me know if I can provide you with anything to investigate the problem.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-09-12T10:06:02.7133333+00:00

Hi Nguyễn Ngọc Phú (VIN3S-KPTPMVF-PPTPMKDHM), Sorry for the trouble causing here. For me, it seems it need more deeper investigation, what exactly happing here. I would encourage you to have a support ticket for this item so that dedicated team can engage and do deep investigation. Below is the link for creating support request.

https://azure.microsoft.com/en-us/support/create-ticket

Share via

ADF MongoAtlas Source side error when batch size existed

1 answer

Your answer