This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
I have 2 files of different format i.e csv,parquet. I need to validate every row present in csv is exactly same as parquet using ADF pipelines
Is there any way to achieve this using ADF or any workaround using other services on Azure with out writing code?
Hi @Mohammed Sohail ,
Just following up to see if the answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.
yes It helped at some extent of comparing files still trying way to include a new column with status
Use the ADF data flow technique of creating row fingerprints via hashing: https://www.youtube.com/watch?v=Id82NZo9hxM&lc
Hi @Mohammed Sohail ,
Just checking in to see if the above answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Hi @MohammedSohailShroff-5457, the approach to deal this scenario discussed in below threads:
https://docs.devart.com/data-compare-for-sql-server/comparing-data/setting-data-sources.html
https://www.c-sharpcorner.com/UploadFile/akkiraju/find-difference-of-data-between-two-data-sources/
The resources specified are much about SQL, I'll elaborate more for better understanding
I have created a ADF pipeline which copy data from excel and create a parquet for every sheet in excel .After the end of the pipeline, I need to create a status file which comapres every row of excel sheet and parquet file and if both rows are same the a boolean of True or False is returned in status file.
So finally in destination I would have two files one parquet file which is copy of sheet in excel and a status file with one column and rows equivalent to dataset which had a value f True/False based on comparision
Can I acheive this using any way in ADF
FYI : I have written a python code which iterates over every row of file and calculate hash for every row. will be doing this for both files and compare hashes if both hash are same it will return as True corresponding to the row else False.