Share via

data factory csv with text qualifier/ingest copy data log file into sql database

Mrinmoy Gupta 1 Reputation point
2021-09-17T09:28:12.667+00:00

I'm trying to use a copy data activity in a data factory pipeline to ingest "copy data log files" from csv to azure sql database for further analysis. While SSIS can parse the csv files correctly data factory is struggling with the commas contained within the data. Using the same configuration ( comma delimiter & " text qualifier) SSIS works but data factory does not. Any idea how to achieve this? Is there any other way of ingesting the log files into SQL database at all?133065-data-factory-csv-file-issue.pdf

Thanks in advance for the help/

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,645 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee
    2021-09-20T09:25:23.023+00:00

    Hi @Mrinmoy Gupta ,

    Welcome to Microsoft Q&A Platform. Thank you for posting query here.

    I tried creating dataset on csv which has commas in some column by defined column delimiter as , and Quote character as ". Its working perfectly fine for me. Kindly check below gif for same.

    133581-csvdataset.gif

    • Could you please review your csv file is comma contained columns enclosed with double quotes properly or not?
    • Could you please clear schema inside dataset and import schema freshly from connection or store and check, if that helps?

    Please let us know how it goes. Thank you.

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.