Share via

Read CSV from ADLS 2

Emil Mankowski 25 Reputation points
Jun 15, 2023, 2:24 PM

Hi,

In Microsoft CRM application, I have configured data export to DataLake. We have annual file partitioning.

User's image

I'm having trouble reading files in Azure Data Factory. The file structure is read from the first row, while the file may contain records with a different number of columns. In ADF, I'm using a DataSet connected to ADLS2 as the source.

User's image

But in file exists rows with that column but it isn't imported by ADF to schema.

Do you have any idea how to solve this case?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,548 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,365 questions
0 comments
Accepted answer
  1. AnnuKumari-MSFT 34,361 Reputation points Microsoft Employee
    Jun 20, 2023, 2:57 AM

    Hi Emil Mankowski ,

    Welcome to Microsoft Q&A platform and thanks for posting your question here.

    As I understand your issue, you are trying toIt seems that the issue is in source dataset, you have imported the schema for one of the files and since the other file '2019.csv' doesn't have same schema , it is failing due to schema mismatch.

    In case you are using dataflow, I would request you to clear the source dataset schema first and then use 'Allow schema drift' option in the source transformation and see if it works or not.

    Please let me know how it goes. Thankyou

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.