Add additional Row when copying files

Anonymous
2020-07-04T09:02:08.433+00:00

Hello, ADLS has a parquet file, using copy activity to move files from one location to another. When copying I want to add an additional column in the sink. Actual data from Source Plus one column like a header at the top. ![11423-images.png][1] [1]: /api/attachments/11423-images.png?platform=QnA Please recommend me on the best approach.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,111 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,437 Reputation points Microsoft Employee
    2020-07-25T00:10:45.007+00:00

    Hi anonymous user,

    Sorry for the delay in response. Unfortunately there is no out of box feature to add a row while copying files. I would recommend you to please provide a feedback in ADF user voice forum : https://feedback.azure.com/forums/270578-azure-data-factory . All the feedback shared in this forum are actively monitored and reviewed by ADF engineering team. Also please do share the feedback link here once it is created, as it would help others with similar idea to up-vote and/or comment on the suggestion/feedback.

    One possible alternate could be to use Custom activity or a Function activity and write your own logic to achieve this.

    ----------

    Thank you
    Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

    0 comments No comments

3 additional answers

Sort by: Most helpful
  1. ChiragMishra-MSFT 956 Reputation points
    2020-07-06T05:57:49.48+00:00

    Hi @Maaran-7752,

    In addition to copying data from source data store to sink, you can also configure to add additional data columns to copy along to sink. For example:

    • When copy from file-based source, store the relative file path as an additional column to trace from which file the data comes from.
    • Add a column with ADF expression, to attach ADF system variables like pipeline name/pipeline id, or store other dynamic value from upstream activity's output.
    • Add a column with static value to meet your downstream consumption need.

    You can find the following configuration on copy activity source tab:

    copy-activity-add-additional-columns.png

    Ref - Add additional columns during copy

    Hope this helps.

    0 comments No comments

  2. Anonymous
    2020-07-06T13:16:53.543+00:00

    @ChiragMishraMSFT-1092 - Sorry for the confusion, I have to add additional row in the sink.


  3. Anonymous
    2020-07-26T20:36:25.017+00:00