Can ADF dataflow assert errors be captured in monitor?

Question

Hi,

Can ADF dataflow assert errors be captured in monitor?

My actual requirement is :

I'm doing some ETL in ADF. After transformation when data got populated in Sink, I want to validate the data is in correct format against some business rules. Can it be done through ADF dataflow assert? If yes, can I see the errors in ADF monitor?

Thanking you in advance.

Regards,
Partha

Answer 1

Hi @Pratim Das, Partha C ,

Thank you for posting query in Microsoft Q&A Platform.

Yes Assert transformation enables you to build custom rules inside your mapping data flows for data quality and data validation.

Using assert transformation will not make data flow activity or pipeline to fail. Hence, You cannot see them as failures in Azure data factory monitor tab.

Inside Assert transformation we get isError() & hasError() functions that tells us that particular row passed Assert or Failed error. Based on that you can take that rows in to some table or file. And then may be try to read that data and if data present then use Fail activity to make pipeline to fail. But this will only make sure to fail the pipeline alone. I feel taking error rows in to some table and then make a decision of further process may be good idea here.

Please check below video to understand about Assert Transformation and isError() & hasError() functions.
Assert Transformation in Mapping Data Flows in Azure Data Factory

Hope this helps.

----------

Please consider hitting Accept Answer button. Accepted answers help community as well.