pipeline should re-run even if tables are failed due to column mismatch

Ram Babu 45 Reputation points
2023-10-09T15:29:42.72+00:00

Hi Team,

I need to handle the scenario where the structure of the source table changes then pipeline should run for failed tables again using drop the table and create new table.

i have design the pipeline using lookup activity, foreach activity and copy activity for 12 tables. source is salesforce and target is azure synapse dedicate pool. every day pipeline runs successful.

Today 11 tables are successful and 1 table failed due to column mismatch in target table in that pipeline.

i have to add some activities (please advice) to re-load if any tables failed. (like drop the table and auto create table.

Please advice how to design pipeline to avoid if source side structure change scenarios.

Thanks

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,460 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,717 questions
{count} votes