issue converting parquet format to csv in ADF

braxx 441 Reputation points
2021-01-04T15:39:10.007+00:00

Having the issue converting parquet data type to csv in ADF.
I have a pipeline to copy from one blob to another replacing the format of the file from parquet to csv.
Here is a sample of the data:

header1 header2 header3 header4 header5
1107672220 474 Straight 5/8" N/A Y 12/31/9999
1107679364 471 5/8" Y 12/31/9999
1103448910 67 Snacks, Coffee, and Pet Y 6/22/2008

For cleaner view the same but on the screenshot:
53351-capture20.png

The output csv is then formatted incorrectly. It may be broken differently depending on the settings in a sink data set – escape character or quote character. have tried several options but none of them work for me. Here are a few of them:

  1. Data for some rows in 3rd column are combined
    53168-capture21.png
  2. In this one data are split by delimiters where it shouldn’t be. Extra columns were created
    53334-capture22.png
  3. I used different settings, but the output is similar as in 1) 53315-capture23.png

I tried different set of configurations but none of them works.
There are similar threads here none of them clearly indicates the solution.

https://stackoverflow.com/questions/61593352/azure-data-factory-escape-character-and-quote-issue-copy-activity
https://learn.microsoft.com/en-us/answers/questions/115904/azure-data-factory-adf-escape-character-issue-when.html

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,825 questions
{count} votes

Accepted answer
  1. Nasreen Akter 10,791 Reputation points
    2021-01-04T20:39:04.163+00:00

    Hi @braxx ,

    Please try with the following settings. Hope this helps! :)

    53367-csv-1.jpg

    3 people found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.