Issue with the new Salesforce connector V2

Lionel Dsouza 0

Hi,

Previously in the Salesforce(Legacy connector) you could pull all the data for an object but with the new Salesforce OAuth 2.0 connector in ADF, it restricts the data?

How to tackle this issue? Should i run in batches?

How to load data in batches with the new Salesforce connector in Azure Data Factory?

Also the new SOQL doesn't support query : select * from Account; which was supported in the legacy connector

For example, with the Legacy connector for Accounthistory object it could retrieve all the rows : 1,224,726 rows but with the new Salesforce connector only retrieves 285,618 rows.

Smaran Thoomu 11,615 Reputation points Microsoft Vendor

2024-05-23T10:43:16.75+00:00

Hi @Lionel Dsouza

Thanks for the question and using MS Q&A platform.

You're right, with the new Salesforce OAuth 2.0 connector in Azure Data Factory, you can't pull all the data for an object. As far as I know to tackle this issue, you can run the data load in batches. You can use the "Lookup" activity to get the maximum and minimum values of the primary key column of the Salesforce object. Then, you can use the "ForEach" activity to loop through the primary key values in batches and use the "Copy" activity to copy the data in batches.

Another option is you can use the Salesforce Bulk API to pull data from Salesforce objects in Azure Data Factory, but it has few limitations. Regarding the SOQL query, the new Salesforce OAuth 2.0 connector doesn't support the select * from Account query, which was supported in the legacy connector. Instead, you need to specify the columns you want to retrieve in the query. For example, "select Id, Name, CreatedDate from Account".

For more Information refer this article - Copy data from and to Salesforce

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Lionel Dsouza 0 Reputation points

2024-05-23T11:07:54.6733333+00:00
Hi Smaran,

Thank you for the options :

"Lookup" activity to get the maximum and minimum values of the primary key column of the Salesforce object. Then, you can use the "ForEach" activity to loop through the primary key values in batches and use the "Copy" activity to copy the data in batches. Do you have any example done for this?

Salesforce Bulk API to pull data from Salesforce objects in Azure Data Factory. How can I implement this? Is this not the Legacy connector?
Lionel Dsouza 0 Reputation points

2024-05-23T11:33:32.8033333+00:00

Hi Smaran,For Option 1 , do you have an example implementation?

For Option 2, do you mean using the Legacy Salesforce connector which uses Bulk API?
Smaran Thoomu 11,615 Reputation points Microsoft Vendor

2024-05-23T14:10:53.3666667+00:00

@Lionel Dsouza Thank you for your query. Unfortunately, we don't have example implementations for loading data in batches using the "Lookup", "ForEach", and "Copy" activities in Azure Data Factory. Additionally, we can't reproduce the issue because it's related to a third-party service, and we don't have control over the Salesforce OAuth 2.0 connector. However, you can follow the documentation article I provided earlier. Please let us know if you have any further questions or concerns. Thank you.
Smaran Thoomu 11,615 Reputation points Microsoft Vendor

2024-05-24T03:59:25.3433333+00:00

@Lionel Dsouza We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Lionel Dsouza 0 Reputation points

2024-05-24T07:17:45.19+00:00

Hi Smaran,

I am currently checking ways to implement this. If I have a solution, I will post. If you come across any other solution, let me know.

According to the other post https://learn.microsoft.com/en-us/answers/questions/1669230/salesforce-v2-link-service-doenst-work-error-messa , there are some bugs identified with the new Salesforce Connector V2.

1 answer

Smaran Thoomu 11,615 Reputation points Microsoft Vendor

2024-05-27T04:58:34.33+00:00

Hi @Lionel Dsouza

Thank you for providing additional details.

It has been identified that there are some bugs with the new Salesforce connector V2 in Azure Data Factory. As a workaround, you can try using the legacy Salesforce connector until the issues with the new connector are resolved. Please refer to Bhargava's suggestions, which were shared in the same link.

However, it's important to note that using the legacy connector may not be a long-term solution, as it may be deprecated in the future. We recommend keeping an eye on updates from Microsoft and Salesforce regarding the new connector and its issues.

I hope this information is helpful. Please let me know if you have any further questions or concerns.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Issue with the new Salesforce connector V2

1 answer