Error Handling when copy data to Salesforce in Azure Data Factory

Liu, Veronica Siqi 26 Reputation points
2022-05-25T05:11:48.597+00:00

Dear Sir,

I have a question about error handling during Copy data to Salesforce.

The scenario is that I use Copy data Activity to load data into Salesforce, and I get some error message from Salesforce responded in the debug output. But it seems the output content is not a full response from Salesforce.

As we know Salesforce Copy data Activity is based on the REST/Bulk API, it can choose automatically.
I want to catch all the error message and error lines which responded from Salesforce Bulk API, and transform it to a csv, then save to Blob or write into data base. How can I implement this via Azure data factory?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,463 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,661 Reputation points Microsoft Employee
    2022-05-27T09:40:04.863+00:00

    Hi @Liu, Veronica Siqi ,

    Thank you for posting query in Microsoft Q&A Platform.

    As you rightly pointed Salesforce connector is built on top of the Salesforce REST/Bulk API. Based on data size service automatically pics either REST or Bulk API. If data size is huge then it will choose Bulk API for better performance. The API version to choose can be controlled from apiVersion option in linked service. Click here to know more about salesforce connector.

    Usually any error you get in ADF which using copy activity or any other activity can be seen in output JSON of activity execution. You can consider taking required details from output json of copy activity and then passing them to stored procedure to store in a table or storing them in to csv file according to your requirement.

    Below few resources or videos which helps.

    If you want to log error details to csv file then use another copy activity on failure of main copy activity. and In source use some dummy file and add additional column under source tab. For this addition column pass your error details from previous copy activity. Under mappings tab load only required columns as csv file.

    Check this video which explains Add additional columns during copy in Azure Data Factory

    Hope this helps. Please let us know if any further queries.

    -----------------

    Please consider hitting Accept Answer button. Accepted answers help community as well.

    0 comments