how to copy data from bloob storage to a output file by using copy pipeline activity

Question

Hi Team,

I have a requirement in azure data factory, That is in azure blob storage we have 2 files separately named employee_info and Department_info I have to create a output file with the below transformations

1.need Create Pipeline to copy the data from source file to target file.

Transformation to be done like below:

1.Derive age based on DOB
2.format sex like "F" to female and "M" to male
3.If salary is greater than 20000 mark as elgible for promotion

4.Output file should be like : Sno Name DOB Sex description Age Salary Departmentname Eligibility

All the fields are existed in the above files

Kindly help anyone to achieve the above tasks...

Thanks.
Eshwar

Answer 1

Hi @Eshwar Nakkala ,

As you need to do Data Transformation, you will need DataFlow to generate the expected output. You have to the following:

1. Create Link Service to connect with the BlobStorage
2. Create Dataset for different Blob File (e.g., one DataSet for Employee-Info and another Dataset for Department-Info)
3. For each Dataset, import the schema
4. Create a DataFlow with the two Datasets as a sources and do join, derive, filter activity based on your requirements and Sink the required columns to the output file.

Please let me know if you need further assistance. Thanks!

----------

If the response is helpful, please Accept Answer and up-vote it. Thanks!