Dynamic ADF Data Transformation Challenge

Question

I'm presently engaged in an ADF data transformation task utilizing the .NET SDK, employing Data Flow for the transformation process. Within a CSV file containing three columns—First Name, Last Name, and Age—I aim to merge First Name and Last Name into a single column labeled Full Name. However, a challenge arises as the input file is dynamically generated at runtime, with datasets being parameterized. Attached is the Data Flow Diagram illustrating my approach. I've attempted to incorporate parameters for the Concat function, yet encountered obstacles.

I have successfully executed this transformation for an existing file stored in blob storage, as depicted in the accompanying Data Flow Diagram. However, my objective is to achieve the same outcome with a file generated dynamically at runtime. I seek your assistance in resolving this matter.

Answer 1

Hi @Vikranth-2626

Thanks for the question and using MS Q&A platform.

Based on your query, it seems that you are trying to merge the First Name and Last Name columns into a single column labeled Full Name in a CSV file using Azure Data Factory (ADF) Data Flow. However, you are facing challenges as the input file is dynamically generated at runtime, with datasets being parameterized.

To achieve this, you can use the Derived Column transformation in ADF Data Flow to concatenate the First Name and Last Name columns into a new column labeled Full Name. You can use the following expression in the Derived Column transformation:

concat(toString($$COLUMN1), " ", toString($$COLUMN2))

Here, $$COLUMN1 and $$COLUMN2 are the system variables that represent the First Name and Last Name columns, respectively. The toString function is used to convert the columns to strings before concatenation.

To parameterize the input file, you can use the parameters feature in ADF. You can define a parameter for the file path and use it in the source dataset. You can then pass the parameter value at runtime to dynamically generate the file path.

For more information on using parameters in ADF, you can refer below doc:

• Use parameters in Azure Data Factory

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.