Is there a way to read csv file with multiple line headers and drop it in adls gen 2 in parquet format using azure data factory

Question

Is there a way to read csv file with multiple line headers and drop it in adls gen 2 in parquet format using azure data factory

Praveena Uppalapati [ELB] 51

Hi

Could you please suggest a way to read a csv file through sftp with multiple line headers to date lake gen 2 in parquet format using azure data factory.

Thank you
Praveena

KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-15T08:51:42.277+00:00

Hi @Praveena Uppalapati [ELB] ,

Thanks for using Microsoft Q&A forum and posting your query.

To better assist on this requirement, could you please provide/attach a sample of you source file with dummy data and expected output format (output schema format)?

We look forward your response.

Thank you
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-21T19:04:34.057+00:00

Hi @Praveena Uppalapati [ELB] ,

We still have not heard back from you. Could you please share above request details to better assist on this requirement.

Thank you
Praveena Uppalapati [ELB] 51 Reputation points

2021-10-26T07:29:56.92+00:00

Please find the attached sample source file with first four lines being headers and the rows also need to be grouped depending ion the headers.

Thank you.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-27T23:39:01.7+00:00

Hi @Praveena Uppalapati [ELB] ,

Thanks for your response and sharing the requested details. I tried few ways but no success. I'm reaching out to internal team to check if there are any workarounds possible and will keep you posted as soon as I have an update from the team.

Than you for your patience.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-11-12T18:45:13.117+00:00

Hi @Praveena Uppalapati [ELB] ,

Sorry for the delayed response. I tried to implement multiple ways using the sample file provided but no success in ADF. I believe it is currently not possible to achieve your requirement using ADF hence I would recommend to go with a custom code/application to achieve it. You can use that custom code in Custom activity of ADF.

Thank you
Praveena Uppalapati [ELB] 51 Reputation points

2021-11-15T06:17:18.147+00:00

Thank you @KranthiPakala-MSFT for all the investigations. Hope, will get the feature in adf as soon as possible.

Your answer

KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-15T08:51:42.277+00:00

Hi @Praveena Uppalapati [ELB] ,

Thanks for using Microsoft Q&A forum and posting your query.

To better assist on this requirement, could you please provide/attach a sample of you source file with dummy data and expected output format (output schema format)?

We look forward your response.

Thank you
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-21T19:04:34.057+00:00

Hi @Praveena Uppalapati [ELB] ,

We still have not heard back from you. Could you please share above request details to better assist on this requirement.

Thank you
Praveena Uppalapati [ELB] 51 Reputation points

2021-10-26T07:29:56.92+00:00

Please find the attached sample source file with first four lines being headers and the rows also need to be grouped depending ion the headers.

Thank you.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-10-27T23:39:01.7+00:00

Hi @Praveena Uppalapati [ELB] ,

Thanks for your response and sharing the requested details. I tried few ways but no success. I'm reaching out to internal team to check if there are any workarounds possible and will keep you posted as soon as I have an update from the team.

Than you for your patience.
KranthiPakala-MSFT 46,737 Reputation points Microsoft Employee Moderator

2021-11-12T18:45:13.117+00:00

Hi @Praveena Uppalapati [ELB] ,

Sorry for the delayed response. I tried to implement multiple ways using the sample file provided but no success in ADF. I believe it is currently not possible to achieve your requirement using ADF hence I would recommend to go with a custom code/application to achieve it. You can use that custom code in Custom activity of ADF.

Thank you
Praveena Uppalapati [ELB] 51 Reputation points

2021-11-15T06:17:18.147+00:00

Thank you @KranthiPakala-MSFT for all the investigations. Hope, will get the feature in adf as soon as possible.

Share via

Is there a way to read csv file with multiple line headers and drop it in adls gen 2 in parquet format using azure data factory

Your answer