Is there a way to automate data validation using Azure data factory

Question

Is there a way to automate data validation using Azure data factory

Abdusalam Chembakasseri 1

Hi I am part of ETL test team,

Could you please let me know if there is a way to automate data validation using Azure data factory

HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-17T15:56:22.867+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-18T16:17:52.047+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-19T16:20:09.887+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

1 answer

Your answer

HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-17T15:56:22.867+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-18T16:17:52.047+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu
HimanshuSinha-msft 19,486 Reputation points Microsoft Employee Moderator

2021-08-19T16:20:09.887+00:00

Hello @AbdusalamChembakasseri-5431 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

Answer 1

Nandan Hegde 36,146 MVP Volunteer Moderator

Hey,

what type of data validations are you expecting ? Data types or some data level scenarios?
Based on my understanding there is no automated way in ADF for major cases of data validations.
You would have to build your own part like custom logics etc as per your expectations.

One such example is :
You can use the open source tool: Great expectations and call it via Azure functions or Batch job or databricks via ADF. The below link should help :

https://greatexpectations.io/

MadhanKS 1 Reputation point

2022-01-17T07:38:34.317+00:00

Hi Nandan Hedge ,

I am trying to implement Great expectation using Azure functions but we are facing below error .

Result: Failure Exception: OSError: [Errno 30] Read-only file system: './tmp' Stack: File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py", line 405, in _handle__invocation_request invocation_id, fi_context, fi.func, args) File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py", line 612, in _run_sync_func func)(params) File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/GETrigger/main.py", line 54, in main os.chmod('./tmp',0o777)

Can you kindly share code that needs to be used to write the results from great expectations into Blob or any other objects present within Azure.

Regards
Madhan KS

Share via

Is there a way to automate data validation using Azure data factory

1 answer

Your answer