No automatic mapping between source and sink

Obaid UrRehman 166 Reputation points


I have an ADF pipeline with:

  1. A stored procedure - when triggered produces some data and stores it on a database (onprem)
  2. Copy activity which takes data from onprem database and stores it in dataset(blob)

Upon debugging, we found that even if the stored procedure is updated (added a new column to the data) the copy activity does not get the mapping updated. Therefore even if new columns exist the copy activity will only copy the column mentioned in mapping.

Is there a way to automatically update the mapping each time it is called.

We intend to productionalize this pipeline and if the mapping cant is updated automatically, we will have to do releases every time a new column is added in the stored procedure.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,522 questions
0 comments No comments
{count} vote

Accepted answer
  1. Pratik Somaiya 4,126 Reputation points

    Hello @Obaid UrRehman

    I am afraid if this is possible

    Copy Activity is a JSON at the back end, whenever you update the mapping the JSON gets updated, but I don't think you would be able to update the JSON automatically as ADF v2 won't allow that

    Additionally, the Copy activity won't detect a new column addition to a stored procedure, you will have to manually add the new column

    Hence, each time a new column is added, a new release would be needed to accommodate that in the mappings

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Nandan Hegde 27,391 Reputation points MVP

    Hey ,
    Actually you can make the mapping dynamic


    So you can create a lookup table in the database and refer it in the ADF.

    Note: The Lookup table can be auto updated whenever there is any new columns added in SP

    0 comments No comments