How to match and validate structure of multiple files in ADF?

Trapti Ruhela 20

Hi Team,

My usecase is to validate schema of multiple files or structure of multiple files against one structure file in ADF. Can someone help me to solve this problem how can I validate

Manoj Kadam 0 Reputation points

2023-05-07T07:48:18.2033333+00:00

In Azure Data Factory (ADF), you can use the "Validate and Profile Data" feature to match and validate the structure of multiple files. Here's how you can do it:

Create a new pipeline in ADF.

Add a new "Get Metadata" activity to the pipeline.

In the "Get Metadata" activity, select the dataset that contains the files you want to validate.

Under the "Settings" tab, select the "Validate schema" checkbox.

Under the "Settings" tab, select the "Get child items" checkbox if you want to validate the schema of all the files in the folder.

Click on "Preview data" to validate and profile the schema of the files.

In the "Preview data" window, you can see the structure of the files and any validation errors that have been detected.

If there are any errors, you can click on the "Details" link to see more information about the error.

Once you have resolved any errors, you can save and run the pipeline to match and validate the structure of the files.

By using the "Validate and Profile Data" feature in ADF, you can easily match and validate the structure of multiple files. This is a powerful tool that can help you ensure the integrity of your data and prevent any issues that may arise due to data inconsistencies or schema mismatches.

Accepted answer

Suba Balaji 11,186 Reputation points

2023-05-05T17:06:35.1366667+00:00

Hi

You could leverage structure property in get meta data activity and if condition to validate schema.

You could go through this video for reference https://youtu.be/LKjewkeNiJ0

If you want to validate many files against one file you could use the same approach but with a foreach loop activity.

Let us know if you need help anywhere in the process.

Thanks
Please sign in to rate this answer.
Trapti Ruhela 20 Reputation points

2023-05-07T07:46:55.0233333+00:00

Hi Surbhi,

Thanks for the help but I could not be able to validate schema of multiple files via foreach activity. I did not find the way how can I validate structure of many files via foreach activity. Please help
Sign in to comment

How to match and validate structure of multiple files in ADF?

0 additional answers