DFExecutorUserError None.get error only in Pipeline not in debug mode

davidsg 6 Reputation points
2022-12-23T08:38:19.883+00:00

Hello, I'm facing an issue that is driving me mad. I get a DFExecutionUserError None.get without any other information when trying to execute a pipeline in Live mode. This error doesn't happens when I run it outside the live mode or in debug mode.

The pipeline executes a DataFlow that filters some information from a datalake, modifies a column using a Derive Column action and then it sinks it to an API REST.

I have checked both this thread and this thread. I have checked and updated the schema from the source both in the Dataset and in the source. I have also checked as true the Validation Schema mark, so I'm quite sure that there is not a mismatch between the schema of the source and the expected schema. Since the sink is not a database, but an API, there is not an expected schema and the API is not receiving any request.

This is the complete error I receive:
{
"StatusCode": "DFExecutorUserError",
"Message": "Job failed due to reason: None.get",
"Details": "java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:306)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:295)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.microsoft.datafactory.dataflow.MetricsUtility$.getMetricsForSink(MetricsUtility.scala:295)
at com.microsoft.datafactory.dataflow.MetricsUtility$.metricsToPayload(MetricsUtility.scala:463)
at com.microsoft.datafactory.dataflow.AdmsClient.getMonitoringPayloadInternal(AdmsClient.scala:485)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at sca"
}

And this is the output from the run dataflow element in my pipeline:
{
"runStatus": {
"computeAcquisitionDuration": 194181,
"dsl": null,
"profile": null,
"metrics": null,
"executorVersion": "20221220.6"
},
"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (West Europe)",
"billingReference": {
"activityType": "executedataflow",
"billableDuration": [
{
"meterType": "General",
"duration": 0.5193346866666667,
"unit": "coreHour",
"sessionType": "JobCluster"
}
]
},
"reportLineageToPurview": {
"status": "NotReported"
}
}

Thanks in advance for any help you may provide.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,329 questions
{count} votes

1 answer

Sort by: Most helpful
  1. davidsg 6 Reputation points
    2023-01-17T07:53:33.3166667+00:00

    Just for the record. This problema was solved by the MS support after opening a ticket with them. The executorVersion that solved the problem was the 20230109.