Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
Job failed due to reason: at Sink 'SinkX': java.lang.ArrayIndexOutOfBoundsException: 0
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 50%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFD%3C/text%3E%3C/svg%3E)
Francisco Dominguez
386
Reputation points
Good morning,
We've been using our pipeline for quite a while now and it contains a dataflow that parses certain files under a certain criteria from a folder. Since yesterday, we've been receiving a weird error even though we haven't changed anything on the pipeline nor the dataflow for this to happen. The error is the following:
Operation on target ImportTradeX failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'SinkX': java.lang.ArrayIndexOutOfBoundsException: 0","Details":"java.lang.ArrayIndexOutOfBoundsException: 0\n\tat com.microsoft.dataflow.spark.AbortExec.doExecute(AbortExec.scala:30)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:157)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:153)\n\tat org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:181)\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\n\tat org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:178)\n\tat org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:153)\n\tat org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:391)\n\tat com.microsoft.dataflow.spark.FineMetricExec.inputRDDs(MetricExec.scala:26)\n\tat org.apache.spark.sql.execution.ProjectExec.inputRDDs(basicPhysicalOperators.scala:46)\n\tat com.microsoft.dataflow.spark.FineMetricExec.inputRDDs(MetricExec.scala:26)\n\tat org.apache.spark.sql.execution.WholeStageCodegenEx"}
Our dataflow contains a source, an excel file (that is usually empty), a derived column, an asset activity to check for duplicates and two sinks. Using the data preview feature, we see that this error is coming up in the assert activity, when we check for duplicates.
I'm willing to share any details to help me with this issue, but I do not want to paste all the details here, so if you need all the information, I will send it via email once you ask me.
Thanks in advance.
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator2022-03-28T05:18:35.253+00:00 Hello @Francisco Dominguez ,
Thanks for the question and using MS Q&A platform.
As per my understanding your pipeline with dataflow was working fine for a while now without any issues and all of sudden you started seeing above error. Please correct me if I'm wrong.
As you haven't changed anything on your dataflow/pipeline, it sounds like something could be wrong on the product side.
Could you please confirm if you are seeing the error/issue consistently or was it a transient one? If you are still seeing the issue, I would recommend filing a support ticket for deeper investigation. In case if you don't have a support plan, please let us know here so we can see for other options to further troubleshoot this issue.
Thank you
-
Francisco Dominguez 386 Reputation points
2022-03-28T07:35:42.843+00:00 Hi @KranthiPakala-MSFT ,
You are correct, we changed nothing in our pipeline/dataflow. I've been doing some more tests to ensure there's an issue, and I can confirm that the issue appears when the CSV file that's coming from the source is empty (has the headers, but no data other than that). If I use a file that has data, then no error appears.
I'm going to go ahead and open a support ticket from my Azure subscription.
Thank you.
-
Francisco Dominguez 386 Reputation points
2022-03-28T07:58:42.247+00:00 Hi again @KranthiPakala-MSFT ,
Turns out my standard support plan does not cover technical support. Could you please then try to help me to further troubleshoot this, or are the details above enough for you to proceed?
Thank you.
-
Paul Hernandez 686 Reputation points Microsoft Employee2022-04-13T14:36:34.99+00:00 Hi @KranthiPakala-MSFT and @Francisco Dominguez ,
we are experience exactly the same issue. With ticked the option "Ignore nulls" without success.
Have you find the cause or a possible workaroung for this problem?
Thanks in advanced.
Paul Hernandez
-
Francisco Dominguez 386 Reputation points
2022-04-20T06:06:54.987+00:00 I've been in touch with Microsoft support since March 29th. So far, a solution has not arrived, but a mail from 2 days ago told me that the "internal team" is working in this matter with priority, and once they have an update, they will reach back to me.
That's all I can say for now.
Regards,
Francisco. -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 63%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vijendra Shid 1 Reputation point2022-07-07T09:55:41.367+00:00 Getting below error on sink contact table using dataflow(tried both Dataverse and D365 connector), it is working on account table and other custom tables also.
Sign in to comment
Sign in to answer