datachecks

Question

I have now loaded the data into storage account gen2
What kind of checks can I do to ensure the validity of the loaded data?
Is there a list of standard checks to check and for me to go through?

Thank you

Answer 1

Hello @arkiboys

You can do a data validation by following steps:

1) Create External Table in DW and check if the datatypes are matching or to check if there are any NULLs in Non Nullable fields or to check the count of data etc. This check would ensure the data in the data lake is as per requirement

You can find a detail on Step 1 in this article: https://www.soais.com/adls-data-validation/

2) You can leverage ADF to perform these data checks wherein you can get the file's Metadata and then apply conditions to check if the validations pass or fail, detailed steps can be found here: https://www.altisconsulting.com/au/insights/how-to-validate-data-lake-files-using-azure-data-factory/

Answer 2

Hey,

You can use Assert data transformations for data validations . The below link explains multiple scenarios for data validations :
youtube.com/watch?v=_NzWpTRxt0s

You can also use the open source tool: Great expectations and call it via Azure functions or Batch job or databricks via ADF. The below link should help :

https://greatexpectations.io/