Azure data factory copy activity DBNull error oracle to Iceberg conversion self-hosted integration runtime

Question

Hi,

When using a copy activity in a foreach to iterate through a list of table in an on-premise Oracle database and output into Iceberg tables in Azure Blob Gen2, only 26 (or 28 on last run) tables out of 619 are successful in copying. Each failure gives the following error;

ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'iceberg/raw'.,Source=mscorlib,''Type=System.ArgumentException,Message=Object of type 'System.DBNull' cannot be converted to type 'System.String'.,Source=mscorlib,'

It looks to me it like copy activity isn't handling nulls. I do not want to have a dataflow to handle what copy activity should as the idea is simplicity and flexibility. I can't find any documentation that might help.

An additional problem for later is with all the copy activity successes, when looking at the json files written to blob, only ONE table, identified by column naming convention, is present. So it's not even writing metadata for multiple objects. I'm new to Iceberg by I presume foreach isn't acting as expected.

Does anyone have any wisdom on using the new Iceberg output in Azure and handling errors within the pipeline?

Thanks

Mark

Answer 1

Hi MarkDG ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding , you are trying to copy data from Oracle source to Iceberg using copy activity in ADF pipeline but it is failing with the error: "Object of type 'System.DBNull' cannot be converted to type 'System.String'" .

It seems there is no data to be copied in the source tables which is why it is throwing the above error. You can try to add a lookup activity prior to the CopyActivity to check number of total rows you have in the oracle table. Then, use If condition to check if number of rows > 0, then run the copy activity.
Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou