Salesforce Connector - Objects Not Supported Via Bulk API - How do we upgrade??

Question

Microsoft is asking us to update the Salesforce (legacy) connector to the Salesforce connector by October 11. The latter uses Bulk API 2.0 from Salesforce which comes with a number of limitations that the legacy connector does not have.

My organization uses the Salesforce (legacy) connector in Azure Data Factory to load Salesforce data into our SQL data warehouse. We use the Copy Activity for all of our Salesforce pipelines.

One major limitation of the new connector is that there is a list of objects that are not supported by the Bulk API. Salesforce documents the limitation here - https://help.salesforce.com/s/articleView?id=000383508&type=1

How are we supposed to 'upgrade' our Copy Activity tasks to the new connector for these unsupported objects? Many of the objects on the not supported list have been used in our reporting layer and are critical to business. What is Microsoft's solution for this limitation?

Answer 1

Here are steps that help you upgrade your linked service and related queries:

1. Configure the connected apps in Salesforce portal by referring to Prerequisites.
2. Create a new Salesforce linked service and configure it by referring to Linked service properties.
3. If you use SQL query in the copy activity source or the lookup activity that refers to the legacy linked service, you need to convert them to the SOQL query. Learn more about SOQL query from Salesforce as a source type and Salesforce Object Query Language (SOQL).
4. readBehavior is replaced with includeDeletedObjects in the copy activity source or the lookup activity. For the detailed configuration, see Salesforce as a source type.

Differences between Salesforce and Salesforce (legacy)

https://learn.microsoft.com/en-us/azure/data-factory/connector-salesforce?tabs=data-factory#differences-between-salesforce-and-salesforce-legacy

For the unsupported objects, you can use the Salesforce REST API or SOAP API to retrieve the data. These APIs do not have the same limitations as the Bulk API. You can set up custom pipelines in Azure Data Factory to utilize these APIs.

Using REST API:

• Create an HTTP Linked Service in Azure Data Factory to connect to Salesforce.
• Create a Web Activity or Copy Activity to call the REST API endpoint and retrieve the data.

Using SOAP API:

• Similar to the REST API, create an HTTP Linked Service and use a Web Activity or Copy Activity to retrieve data. Using REST API:
  • Create an HTTP Linked Service in Azure Data Factory to connect to Salesforce.
  • Create a Web Activity or Copy Activity to call the REST API endpoint and retrieve the data.