Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Is there a way to read csv file with multiple line headers and drop it in adls gen 2 in parquet format using azure data factory
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 23%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPU%3C/text%3E%3C/svg%3E)
Praveena Uppalapati [ELB]
51
Reputation points
Hi
Could you please suggest a way to read a csv file through sftp with multiple line headers to date lake gen 2 in parquet format using azure data factory.
Thank you
Praveena
Azure Data Factory
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,732 Reputation points Microsoft Employee Moderator2021-10-15T08:51:42.277+00:00 Hi @Praveena Uppalapati [ELB] ,
Thanks for using Microsoft Q&A forum and posting your query.
To better assist on this requirement, could you please provide/attach a sample of you source file with dummy data and expected output format (output schema format)?
We look forward your response.
Thank you
-
KranthiPakala-MSFT 46,732 Reputation points Microsoft Employee Moderator
2021-10-21T19:04:34.057+00:00 Hi @Praveena Uppalapati [ELB] ,
We still have not heard back from you. Could you please share above request details to better assist on this requirement.
Thank you
-
Praveena Uppalapati [ELB] 51 Reputation points
2021-10-26T07:29:56.92+00:00 Please find the attached sample source file with first four lines being headers and the rows also need to be grouped depending ion the headers.
Thank you.
-
KranthiPakala-MSFT 46,732 Reputation points Microsoft Employee Moderator
2021-10-27T23:39:01.7+00:00 Hi @Praveena Uppalapati [ELB] ,
Thanks for your response and sharing the requested details. I tried few ways but no success. I'm reaching out to internal team to check if there are any workarounds possible and will keep you posted as soon as I have an update from the team.
Than you for your patience.
-
KranthiPakala-MSFT 46,732 Reputation points Microsoft Employee Moderator
2021-11-12T18:45:13.117+00:00 Hi @Praveena Uppalapati [ELB] ,
Sorry for the delayed response. I tried to implement multiple ways using the sample file provided but no success in ADF. I believe it is currently not possible to achieve your requirement using ADF hence I would recommend to go with a custom code/application to achieve it. You can use that custom code in Custom activity of ADF.
Thank you
-
Praveena Uppalapati [ELB] 51 Reputation points
2021-11-15T06:17:18.147+00:00 Thank you @KranthiPakala-MSFT for all the investigations. Hope, will get the feature in adf as soon as possible.
Sign in to comment
Sign in to answer