How to add derived column using synapse dataflow where source is azure sql and sink is workspace db (synapse delta table)

Question

I have created dataflow where source is Azure SQL Server. I need to add the derived column and load data into workspace DB. I am following metadata driven approach in which the select query for source to pull the data from data source, sink database name and sink table name is stored in azure sql table. Using pipeline lookup activity I am reading the metadata and inside for each loop the dataflow will be executed to which parameters are passed.

Here is the error :

Operation on target ForEach failed: Activity failed because an inner activity failed; Inner activity name: DataFlow_SourceToLanding, Error: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'sink1': org.apache.spark.sql.AnalysisException: The column number of the existing table default.phaseone(struct<>) doesn't match the data schema(struct<Id:int,Name:string,ETLId:string>);","Details":"org.apache.spark.sql.AnalysisException: The column number of the existing table default.phaseone(struct<>) doesn't match the data schema(struct<Id:int,Name:string,ETLId:string>);\n\tat

Answer 1

When you've selected None for the table action, then the schemas must match. If the schema differs from the database you've already defined as the Workspace DB, you'll need to recreate the table or create a new table.