how to copy data from bloob storage to a output file by using copy pipeline activity

Eshwar Nakkala 106 Reputation points
2021-02-03T13:42:26.8+00:00

Hi Team,

I have a requirement in azure data factory, That is in azure blob storage we have 2 files separately named employee_info and Department_info I have to create a output file with the below transformations

1.need Create Pipeline to copy the data from source file to target file.

Transformation to be done like below:

1.Derive age based on DOB
2.format sex like "F" to female and "M" to male
3.If salary is greater than 20000 mark as elgible for promotion

4.Output file should be like : Sno Name DOB Sex description Age Salary Departmentname Eligibility

All the fields are existed in the above files

Kindly help anyone to achieve the above tasks...

Thanks.
Eshwar

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,774 questions
0 comments No comments
{count} votes

Accepted answer
  1. Nasreen Akter 10,761 Reputation points
    2021-02-03T14:29:10.207+00:00

    Hi @Eshwar Nakkala ,

    As you need to do Data Transformation, you will need DataFlow to generate the expected output. You have to the following:

    1. Create Link Service to connect with the BlobStorage
    2. Create Dataset for different Blob File (e.g., one DataSet for Employee-Info and another Dataset for Department-Info)
    3. For each Dataset, import the schema
    4. Create a DataFlow with the two Datasets as a sources and do join, derive, filter activity based on your requirements and Sink the required columns to the output file.

    Please let me know if you need further assistance. Thanks!

    ----------

    If the response is helpful, please Accept Answer and up-vote it. Thanks!

    2 people found this answer helpful.

0 additional answers

Sort by: Most helpful