Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
I have some good news. A fix should be rolled out to all regions today. This was a recent bug.
Could you please test again now?
dataflow job failed - None.get
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,711
Reputation points
hello,
I am executing a dataflow from a pipeline...
dataflow on its own which writes to delta parquet in azure storage works fine in debug when testing it on its own.,
The dataflow, simply joins .csv files together and write to delta...
When I execute it from a pipeline, I see this message and not sure what is causing this error.
Any thoughts please? thank you
Error details
Error code
DFExecutorUserError
Failure type
User configuration issue
Details
Job failed due to reason: None.get
Source
Pipeline
pipeline1
Data flow
df_testflow
Monitor
Data flow activity
Data flow1
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT 26,161 Reputation points2022-12-14T21:26:35.53+00:00 Hi again @arkiboys . This one looks weird. So as I understand the dataflow works fine in a debug run, but a triggered pipeline fails with that error?
Hmm, well the difference between a debug's run and a trigger's run, is where the instructions are fetched from. Can you try doing a publish? Maybe the live version isn't the same as the displayed version.
If it had never been published, I would expect a different error message. None.get means something important is missing. Does this error happen every time or just once?If this was just a mistake in the data flow script or data I would expect a longer error message. Something so short smells like a problem closer to the runtime.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 28.999999999999996%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
davidsg 6 Reputation points2022-12-20T14:33:07.997+00:00 I have the exact same problem. The dataflow works flawlessly in debug mode, wether it's live mode or not. Only stops working when the pipeline is run and only with that dataflow. Same error message.
{
"StatusCode": "DFExecutorUserError",
"Message": "Job failed due to reason: None.get",
"Details": "java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:306)
at com.microsoft.datafactory.dataflow.MetricsUtility$$anonfun$getMetricsForSink$5.apply(MetricsUtility.scala:295)
at scala.collection.immutable.List.foreach(List.scala:392)
at com.microsoft.datafactory.dataflow.MetricsUtility$.getMetricsForSink(MetricsUtility.scala:295)
at com.microsoft.datafactory.dataflow.MetricsUtility$.metricsToPayload(MetricsUtility.scala:463)
at com.microsoft.datafactory.dataflow.AdmsClient.getMonitoringPayloadInternal(AdmsClient.scala:485)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at com.microsoft.datafactory.dataflow.AdmsEventListener$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$8$$anonfun$apply$1.apply(AdmsEventListener.scala:90)
at sca"
}
Sign in to comment
Answer accepted by question author
-
MartinJaffer-MSFT 26,161 Reputation points
2022-12-22T02:03:24.35+00:00 -
davidsg 6 Reputation points
2022-12-22T08:52:10.407+00:00 Already tested but I keep receiving the same error. I suppose the fix hasn't rolled out yet to my region, so I will keep testing during the day. Using the Auto Resolve IntegrationRuntime from West Europe.
The executor version is: "20221220.6"
-
MartinJaffer-MSFT 26,161 Reputation points
2022-12-27T18:08:11.467+00:00 If it still isn't working, then there must be another cause @davidsg .
I'd recommend opening a support ticket so someone can look into the logs. If you do not have a support plan, I can give you a 1-time free support ticket.
-
davidsg 6 Reputation points
2022-12-30T15:35:00.457+00:00 I don't think we have a support plan so a free support ticket would be appreciated, but I will need to consult my superior about privacy concerns first. Since we are in the holiday season and a lot of us are not working right now It may take some time until I can use it.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 5%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E4%3C/text%3E%3C/svg%3E)
42131580 0 Reputation points2024-01-25T09:33:58.87+00:00 Hi, I have the same issue. My pipeline runs every hour (pulling lastmodified files within last hour) and works fine, but fails several times per day, mostly during night. Is it possible it's due to no new blob files in storage, or any other reason? Thanks,
Sign in to comment -
3 additional answers
Sort by: Most helpful
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 43%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shreyash Choudhary 126 Reputation points2022-12-29T09:11:25.187+00:00 I am also facing the same error, please provide solution
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 34%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
siuwaimok 0 Reputation points2023-01-12T11:29:04.0466667+00:00 Hi,
I got the similar error as well, any solution?
Thanks
-
MartinJaffer-MSFT 26,161 Reputation points
2023-01-17T17:59:52.6966667+00:00 If you are still getting the error, please open a support ticket with Microsoft.