Share via

Copy Activity Fails while copying Multiple JSON Files in ADLS to Azure SQL DB

Suchit Kumar Battula 6 Reputation points
2022-03-31T20:03:28.737+00:00

I have Multiple JSON Files in ADLS GEN2 ( 100s ) ,when there is no Error File the ADF V2 Copy Activity, Copies Entire Files from ADLS Gen2 to Azure SQL DB.
But when there is one error File in the List then the Entire Copying gets failed.
I need to Skip the Error File and Process the Remaining Files.
I could not find a solution, so raising question here. Any Quick Suggestion / help would be highly appreciated.

Additional Info

  1. I am using User Defined Table Type
  2. ADLS Gen2 as Source
  3. Azure SQL DB as Destination
  4. Copy Activity - Source Files are list of JSON Files ( Wild Card ) & Sink is Azure SQL DB ( Stored Procedure)

Thanks,
SK

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,750 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Pratik Somaiya 4,201 Reputation points
    2022-04-01T04:18:34.007+00:00

    Hello @Suchit Kumar Battula

    If you have the files in a list then you can use ForEach activity to loop through that list and execute the copy activity

    In this case, if one of the file has an error then copy would fail only for that file but would succeed for the rest

    You just need to parse the list using ForEach activity