Data flow Output metric Status as substatus instead of runstatus resulting error while capturing metrics for status using runstatus column for logging

Hanuman Devineni 51

Getting Status as Substatus instead of runstatus for DataFlow in Azure data factory, We are using this column to log the status of the pipelines for all data flows & pipelines, Few pipelines are getting failed for the same and getting fixed automatically for next run.

Failure message: The expression 'String(if(contains(activity('ADFPL__df').output.runStatus.metrics,'MapInsert'),if(contains(activity('ADFPL_df').output.runStatus.metrics.MapInsert.sources,'SourceStage'),activity('ADFPL_df').output.runStatus.metrics.MapInsert.sources.SourceStage.rowsRead,0),0))' cannot be evaluated because property 'runStatus' doesn't exist, available properties are 'SubStatus, ComputeAcquisitionDuration, RecordTimestamp, MetricsPayload, effectiveIntegrationRuntime, executionDuration, durationInQueue'.

Matt 5 Reputation points

2024-05-23T20:18:17.8133333+00:00

I've started to see this occurring too in the last week or so. Hasn't occurred before that. I am finding it happens on about every 1 in 10 dataflow executions, and i cannot see a pattern between time between completion of the dataflow, and the starting of the following set variable activity.
Hanuman Devineni 51 Reputation points

2024-05-23T21:13:36.95+00:00

Was there any change in Output properties after Spark Cluster upgrading flow from 3.1 to 3.3

Observe that Runstatus is replaced substatus
Sudipta Kumar Dutta 6 Reputation points

2024-05-24T08:07:16.5433333+00:00

I have started seeing this error randomly for multiple data flows in our different ADF environments from May 22, 2024 evening.
Hanuman Devineni 51 Reputation points

2024-05-24T14:14:44.6933333+00:00

Thanks for response, next run is successful, however issue is sporadic.

Will update, if it happens again.
AnnuKumari-MSFT 33,401 Reputation points Microsoft Employee

2024-06-05T15:55:57.7466667+00:00

Hi Hanuman Devineni ,

Just following up to check if the issue got resolved and not appearing again , kindly consider accepting the answer by clicking on Accept answer button. Thankyou

Accepted answer

AnnuKumari-MSFT 33,401 Reputation points Microsoft Employee

2024-05-24T05:34:22.9033333+00:00

Hi Hanuman Devineni ,

Thankyou for using Microsoft Q&A platform and thanks for raising your query here.

This issue is an old issue which has resurfaced again due to outage in few regions. Backend service was restarted in impacted regions . As per the latest update from the team , the issue has been mitigated now and 'runstatus' should be coming as the output property.

Could you please retry now from your end and see if it is working or the error still persists.

Thankyou
Please sign in to rate this answer.

1 person found this answer helpful.
AnnuKumari-MSFT 33,401 Reputation points Microsoft Employee

2024-05-28T08:03:14.2033333+00:00

Hi Hanuman Devineni ,

Just following up to seek an update on the issue. Do you happen to see the issue appearing again after that? If not, kindly consider accepting the answer by clicking on Accept answer button. Thankyou

John Bosch 51 Reputation points

2024-06-23T23:14:06.33+00:00

You might need to look at this again as I just had the error again. This happened to us for a while around the date mentioned above, and happened again over the weekend just gone.

{ "SubStatus": "Success", "ComputeAcquisitionDuration": 2859, "RecordTimestamp": "2024-06-21 08:47:41.023", "MetricsPayload": "xxxxx", "effectiveIntegrationRuntime": "ManagedVnetIntegrationRuntime (Australia East)", "executionDuration": 78, "durationInQueue": { "integrationRuntimeQueue": 0 } }
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Data flow Output metric Status as substatus instead of runstatus resulting error while capturing metrics for status using runstatus column for logging

0 additional answers

Your answer