No automatic mapping between source and sink

Question

Hi,

I have an ADF pipeline with:

1. A stored procedure - when triggered produces some data and stores it on a database (onprem)
2. Copy activity which takes data from onprem database and stores it in dataset(blob)

Upon debugging, we found that even if the stored procedure is updated (added a new column to the data) the copy activity does not get the mapping updated. Therefore even if new columns exist the copy activity will only copy the column mentioned in mapping.

Is there a way to automatically update the mapping each time it is called.

We intend to productionalize this pipeline and if the mapping cant is updated automatically, we will have to do releases every time a new column is added in the stored procedure.

Answer 1

Hello @Obaid UrRehman

I am afraid if this is possible

Copy Activity is a JSON at the back end, whenever you update the mapping the JSON gets updated, but I don't think you would be able to update the JSON automatically as ADF v2 won't allow that

Additionally, the Copy activity won't detect a new column addition to a stored procedure, you will have to manually add the new column

Hence, each time a new column is added, a new release would be needed to accommodate that in the mappings

Answer 2

Hey ,
Actually you can make the mapping dynamic

So you can create a lookup table in the database and refer it in the ADF.

Note: The Lookup table can be auto updated whenever there is any new columns added in SP