Copy activity Error when copy data from Oracle to Azure Databricks. Type=System.NullReferenceException, Message=Object reference not set to an instance of an object

Jiratchaya Masaowapak 0 Reputation points
2023-05-30T06:01:59.34+00:00

I triggered parallel pipeline runs to copy data from Oracle to the same target table using the same pipeline and configuration, but with different queries.

Let's say the first script is 'SELECT * FROM table_a WHERE data_dt = '2022-01-01', and the second script is 'SELECT * FROM table_a WHERE data_dt = '2021-01-01'.

One pipeline run succeeds, but the other fails during the process of copying data from Azure Data Lake Storage to Azure Databricks Delta Lake, with the following error:

'Type=System.NullReferenceException, Message=Object reference not set to an instance of an object., Source=Microsoft.DataTransfer.Connectors.AzureDatabricksDeltaLake.'

These are more details about the error from the output details of the copy activity:

"errors": [
	{
		"Code": 11000,
		"Message": "'Type=System.NullReferenceException,Message=Object reference not set to an instance of an object.,Source=Microsoft.DataTransfer.Connectors.AzureDatabricksDeltaLake,'",
		"EventType": 0,
		"Category": 5,
		"Data": {},
		"MsgId": null,
		"ExceptionType": null,
		"Source": null,
		"StackTrace": null,
		"InnerEventInfos": []
	}
]

How can I solve it?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,426 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,182 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Bhargava-MSFT 29,041 Reputation points Microsoft Employee
    2023-06-02T23:46:46.8333333+00:00

    Hello Jiratch M,

    Please correct me if my understanding is wrong. You are running a parallel pipeline(two copy activities?) to copy data from Oracle to the same target table using the same pipeline and queries.

    One pipeline run succeeds, but the other fails during the process of copying data from Azure Data Lake Storage to Azure Databricks Delta Lake.

    Is this intermittent or failing whenever you run the copy activity?

    As per the error message, there is an NullReferenceException occurring during the copy process. This error indicates that an object reference is not set to an instance of an object, meaning that a variable or object expected to have a value is null.

    Please try manually splitting the copy activity and check the exception.

    And please check for any null values or missing mappings that could cause the NullReferenceException.

    I hope this helps. Please let us know if you have any further questions.