Copy activity Error when copy data from Oracle to Azure Databricks. Type=System.NullReferenceException, Message=Object reference not set to an instance of an object

Question

I triggered parallel pipeline runs to copy data from Oracle to the same target table using the same pipeline and configuration, but with different queries.

Let's say the first script is 'SELECT * FROM table_a WHERE data_dt = '2022-01-01', and the second script is 'SELECT * FROM table_a WHERE data_dt = '2021-01-01'.

One pipeline run succeeds, but the other fails during the process of copying data from Azure Data Lake Storage to Azure Databricks Delta Lake, with the following error:

'Type=System.NullReferenceException, Message=Object reference not set to an instance of an object., Source=Microsoft.DataTransfer.Connectors.AzureDatabricksDeltaLake.'

These are more details about the error from the output details of the copy activity:

"errors": [
	{
		"Code": 11000,
		"Message": "'Type=System.NullReferenceException,Message=Object reference not set to an instance of an object.,Source=Microsoft.DataTransfer.Connectors.AzureDatabricksDeltaLake,'",
		"EventType": 0,
		"Category": 5,
		"Data": {},
		"MsgId": null,
		"ExceptionType": null,
		"Source": null,
		"StackTrace": null,
		"InnerEventInfos": []
	}
]

How can I solve it?

Answer 1

Hello Jiratch M,

Please correct me if my understanding is wrong. You are running a parallel pipeline(two copy activities?) to copy data from Oracle to the same target table using the same pipeline and queries.

One pipeline run succeeds, but the other fails during the process of copying data from Azure Data Lake Storage to Azure Databricks Delta Lake.

Is this intermittent or failing whenever you run the copy activity?

As per the error message, there is an NullReferenceException occurring during the copy process. This error indicates that an object reference is not set to an instance of an object, meaning that a variable or object expected to have a value is null.

Please try manually splitting the copy activity and check the exception.

And please check for any null values or missing mappings that could cause the NullReferenceException.

I hope this helps. Please let us know if you have any further questions.