ADF - Capture the error message ( or exception error) when a data flow activity failed.

Question

I am trying to build a success and failure email notification in Azure Data Factory. My structure is a main pipeline with some "Execute Pipeline" activities, each of these contains "Copy Data" and "Data flow" activities. The idea is capture those "Executed Pipeline" succes or failed status, and for failed ones , also the error message.

For "Copy Data" works good , when it fails I capture the error message with <@activity('activity').error.message> for a specific run id that is contained in output json file.

For "Data Flow" doesn't work, it seems the error is not in the output json file that generates, but it is in the pipeline details thrown as an exception error.

How can I access to this dataflow exception error message in Pipeline Details?

]1

Many thanks for your help, any hints are most welcome.

Answer 1

Hello @Vicent Dolz and welcome to Microsoft Q&A.

I tested what you described. I made the simplest DataFlow I could think of, and added a divide-by-zero in parameter to cause a failure. Then I did debug run.

The error message doesn't look too much different.
So I used a Set Variable activity to capture the raw DataFlow activity details. After cleaning it up, it looks like below.

{  
	"PipelineName": "FailDataFlow",  
	"PipelineRunId": "XXXXXXXXXXXXX",  
	"JobId": "XXXXXXXX",  
	"ActivityRunId": "XXXXXXX",  
	"ExecutionStartTime": "XXXXXX",  
	"ExecutionEndTime": "XXXXXXXXX",  
	"Status": "Failed",  
	"Error": {  
		"message": "{\"StatusCode\":\"DFExecutorUserError\",\"Message\":\"Job failed due to reason: at (Line 2/Col 1): Parameter 'parameter1' default value does not match datatype specified\",\"Details\":\"at (Line 2/Col 1): Parameter 'parameter1' default value does not match datatype specified\"}",  
		"failureType": "UserError",  
		"target": "Data flow1"  
	},  
	"Output": {  
		"runStatus": {  
			"computeAcquisitionDuration": 8109,  
			"dsl": null,  
			"profile": null,  
			"metrics": null,  
			"executorVersion": "XXXXXXX"  
		},  
		"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)",  
		"billingReference": {  
			"activityType": "executedataflow",  
			"billableDuration": [  
				{  
					"meterType": "General",  
					"duration": 0.045342982444444446,  
					"unit": "coreHour",  
					"sessionType": "InteractiveCluster"  
				}  
			]  
		},  
		"reportLineageToCatalog": {  
			"status": "NotReported"  
		}  
	},  
	"ExecutionDetails": {  
		"integrationRuntime": [  
			{  
				"name": "DefaultIntegrationRuntime",  
				"type": "Managed",  
				"location": "East US"  
			}  
		]  
	},  
	"StatusCode": 400,  
	"ExecutionStatus": "Fail",  
	"Duration": "00:00:33.1347714",  
	"RecoveryStatus": "None"  
}  

One notable thing I did have to do, was connect the Set Variable activity to the Dataflow activity by a blue on-completion dependency. If I had used a regular green on-success dependency, the Set Variable activity would not have triggered, and the error only shown up at the pipeline.

To get the error message you would want to use like

@{activity('Data flow1').Error.message}