Query loading file

Question

Query loading file

Niren Adhikary 96

Hello, is it possible to load files into delta table in azure environment from an output file .csv pushed into datalake where the columns are not static and dynamically changing everytime.

Vinodh247 35,221 Reputation points MVP Volunteer Moderator

2023-06-19T13:01:06.0733333+00:00

Hi,

Thanks for reaching out to Microsoft Q&A.

You can try schema evolution option or go with traditional table-driven approach in which you can define the column metadata table and create tables based on that.

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.
Niren Adhikary 96 Reputation points

2023-06-19T13:14:41.95+00:00

Could you explain more? We will be getting new columns everytime in the file.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-06-21T18:36:59.44+00:00

Niren Adhikary

I am checking to see if you have any further questions here.

1 answer

Your answer

Vinodh247 35,221 Reputation points MVP Volunteer Moderator

2023-06-19T13:01:06.0733333+00:00

Hi,

Thanks for reaching out to Microsoft Q&A.

You can try schema evolution option or go with traditional table-driven approach in which you can define the column metadata table and create tables based on that.

Please Upvote and Accept as answer if the reply was helpful, this will be benefitting the other community members who go through the same issue.
Niren Adhikary 96 Reputation points

2023-06-19T13:14:41.95+00:00

Could you explain more? We will be getting new columns everytime in the file.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-06-21T18:36:59.44+00:00

Niren Adhikary

I am checking to see if you have any further questions here.

Answer 1

Niren Adhikary

You can use ADF mapping dataflow to copy data to and from a delta lake stored in Azure data lake using the delta format. Delta is only available as an inline dataset and, by default, doesn't have an associated schema.

High level steps:

create linked service and dataset that connect to your data source (data lake) where csv is stored

delta table as sink

To get column metadata, click the Import schema button in the Projection tab. This will allow you to reference the column names and data types specified by the corpus. To import the schema, a data flow debug session must be active and you must have an existing CDM entity definition file to point to

To use ADF to load files into a delta table where the columns are not static and dynamically changing every time, you can use the delta source script example provided in the below document. The delta source script allows you to specify the column names and data types for the source data. You can modify the script to match the column names and data types in your .csv file.

I hope this helps! Let me know if you have any further questions.

Please see the below document and demonstration video in the document.

https://learn.microsoft.com/en-us/azure/data-factory/format-delta

I hope this helps. Please let us know if you have any further questions.

Share via

Query loading file

1 answer

Your answer