question

ewinkiser avatar image
0 Votes"
ewinkiser asked HimanshuSinha-MSFT commented

Need to add a first row in my Sink CSV File....Urgent, needed for QA please

Hi! @nasreen-akter @MartinJaffer-MSFT I have a real challenge here, at least for me.

We found an issue in QA; I need to add a first row (Row #1) to an already existing CSV file which is a sink of a long pipeline. 81197-screenshot-2021-03-24-112823.jpg

The first row looks something like this:
05, ,2021-01-25, ,BMECOL, ,750

How can I insert this in ADF as the first row to my already existing CSV file? It must be on the first row. I have googled until my fingers hurt :-). This is urgent to us. Thanks as always!
Mike Kiser




azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarkKromer-MSFT avatar image
0 Votes"
MarkKromer-MSFT answered ewinkiser commented

Add a data flow activity to your pipeline. Inside the Data Flow, add 2 sources: 1 for each file. Add a Union transformation to combine the 2 files.

In order to get just the first row from one of the files, add a Surrogate Key transform after your source. Then use Filter to pick row 1.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MarkKromer-8019 , sorry, I should have mentioned; we don't have Data Flows in the Government Cloud and really restricts my development. Do you have any other suggestions? Thanks! Mike Kiser

0 Votes 0 ·
nasreen-akter avatar image
0 Votes"
nasreen-akter answered HimanshuSinha-MSFT commented

Hi @ewinkiser,

I could be wrong, but I think as you do not have DF in the Government Cloud yet, it would not be easy doing this in the pipeline. You could try:

option#1: create an Azure Function to do the merge and call this activity at the end of your existing pipeline
option#2: if the header line has the same number of columns like the items from line#2, you could try to do the following (main idea is to merge multiple csv files into one):
a. create another csv file with the header information (number of columns should be same)
b. create a function to write the filename in sequence to a txt file for example (header1, csv1, ...)
c. add a copyActivity pointing to the folder as a Source Dataset. In the Source tab, select File path Type --> List of files and pointing to the txt file you just created with the correct file order sequence. Also, put Max concurrent connections --> 1.

81529-img28.jpg

Hope this helps! Thanks! :)






img28.jpg (43.3 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

adding @NandanHegde-7720, @VaibhavChaudhari and @HimanshuSinha-MSFT. They have very good understanding in ADF :)

0 Votes 0 ·

Hello @ewinkiser ,

Apoloziges for the delayed response , did you got a resolution to this ?

Thanks
Himanshu

0 Votes 0 ·