Is it necessary to have same format of json for flatting the file ?

Question

So my question is how can i flat these file in single ADF because i a have 1700 total files and facing this issue.

Answer 1

Hello @Aridian Technologies,

Thanks for reaching out on this forum,

Looks like the source data is different from time to time (Data inconsistency)?

Are you sure the Source data format or the schema will not change in future?

As you might have already came to know that there is no Out-Of-the-Box solution for this scenario and we have to handle this by a custom solution!!!

For example, you might be able to use a conditional expression to check the type of shipment and transform it into a consistent format!

Preprocess the Data: You might need to preprocess the files to make them consistent before loading them into ADF. This could involve writing a script that goes through all the files, checks the type of shipment, and converts it into a consistent format. This would likely involve converting all shipment attributes into arrays for consistency.

Use a Stored Procedure activity to process the data in the staging table and insert it into your final destination table. You can use SQL queries or stored procedures to extract and transform the data as needed.

You can then use GetMetadata activity and IfCondition activity to determine the schema of the source file and execute the appropriate activity.

In the end, inconsistent data can pose a serious challenge. Still, you could potentially use a combination of Flatten, Derived Column and other transformations to wrangle the data into the form you need.