Change column names and Precision value according to mapping dataset

Krishnamohan Nadimpalli 406

I have a couple of datasets, one is the original dataset(dataset1) and other is mapping dataset(dataset2). I want to get headers changed and change precision value

User's image

No changes to first 3 columns and rest of them should be taking from mapping dataset(dataset2)

User's image

Amira Bedhiafi 23,011 Reputation points

2023-08-30T11:05:38.1566667+00:00
You need to follow these steps :

Set up Azure Data Factory:

Create an instance of Azure Data Factory if you haven't done so.

Set up linked services to connect to your source and destination.

Create datasets for the source (dataset1 and dataset2) and the output dataset.

Create a Data Flow:

In the ADF portal, create a new data flow.

Read Datasets:

Add two source transformations to read from dataset1 and dataset2.

Rename Columns:

Join the main dataset (dataset1) with the mapping dataset (dataset2) based on the TAG column.

Use a Derived Column transformation to conditionally rename columns based on the mapping from the joined dataset. You'll want to use expressions to replace column names with the corresponding SHORT value.

Adjust Precision:

For each of the columns that need precision adjustment, use a Derived Column transformation to divide or multiply the value by the necessary power of 10 based on the PRECISION value.

For example, for the column U: (same for the remaining columns) : ```pseudo

if(PRECISION == 1) then U / 10 else U
Krishnamohan Nadimpalli 406 Reputation points

2023-08-30T15:59:21.6833333+00:00

Hi @Amira Bedhiafi

I understand the above, but how do we join datasets, and I need to loop through Columns and it needs to be taken dynamically.
Krishnamohan Nadimpalli 406 Reputation points

2023-08-31T17:33:59.3166667+00:00

I was able to make changes for changing the column names using Cache Sink using following snippet.

But I would still require assistance in derived column transformation with column patterns which would take column name as it is from incoming and value of column (incoming_value/precision_from_cachedsink)

Share via

Change column names and Precision value according to mapping dataset

Your answer