Data Lake Old files are not modified after modifying Metadata

Anshuman Chowdhury (NVS EXT) 20 Reputation points
2023-12-19T08:04:59.4233333+00:00

Hi All,

We configured data lake gen2, three months back. After that we modify source table in D365.

Let say earlier my table contains 57 field. we added 3 new fields, now fields count is 60. CDM Json file also updated. Latest CSV file also created with 60 fields.

But Old CSV files are available with 57 field. This creates a problem to sync this data to a separate program.

Need your suggestion.

Regards

Anshuman

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,540 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,541 Reputation points
    2023-12-20T06:57:06.9733333+00:00

    @Anshuman Chowdhury (NVS EXT) - Thanks for the question and using MS Q&A platform.

    It seems like you have modified the metadata of your source table in D365 and added three new fields, resulting in an updated CDM JSON file and a new CSV file with 60 fields. However, the old CSV files are still available with 57 fields, which is causing issues with syncing the data to a separate program.

    Based on the information provided, it is not clear how the old CSV files were generated and where they are stored. However, it is possible that the old CSV files were generated before the metadata was updated in D365, and therefore do not contain the new fields.

    To resolve this issue, you may need to regenerate the old CSV files with the updated metadata. You can do this by running the data export process again with the updated metadata, which should generate new CSV files with all 60 fields. Once you have the new CSV files, you should be able to sync the data to your separate program without any issues.

    If you are using Azure Data Factory to export data from D365 to Data Lake Gen2, you can modify the existing pipeline to include the new fields and run the pipeline again to generate the new CSV files. Alternatively, you can create a new pipeline with the updated metadata and run it to generate the new CSV files.

    I hope this helps! Let me know if you have any further questions.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.