I am using the HubSpot source connector in Azure Data Factory for Companies and Contacts and associated properties. I have a "Copy Data" activity that references the "HubSpot Contact" dataset in entirety, using "Table", not query (so no WHERE clause) I have over 7,000 contacts in my HubSpot instance, however when the source connector gets the data from HubSpot, it only gets 5,900 records. This has been happening ongoing for a number of weeks now (see screenshot below I've looked at the source input for the job and it always returns 5,900 rows read- which obviously then only writes 5,900 to my sink (SQL server in my instance) NOTE: I'm also downloading companyProperties (79K rows) and contactProperties (470K rows). These are downloading fine and are not capping at 5,900 records Is there something specifically that needs to be set in ADF or HubSpot to allow more than 5,900 contacts to be downloaded?

Is there a limitation of the HubSpot Source connector for Contact records to 5,900 records?

Grant Barrington 0

I am using the HubSpot source connector in Azure Data Factory for Companies and Contacts and associated properties.

I have a "Copy Data" activity that references the "HubSpot Contact" dataset in entirety, using "Table", not query (so no WHERE clause)

Screenshot 2024-07-25 090102

I have over 7,000 contacts in my HubSpot instance, however when the source connector gets the data from HubSpot, it only gets 5,900 records. This has been happening ongoing for a number of weeks now (see screenshot below Screenshot 2024-07-25 085653

I've looked at the source input for the job and it always returns 5,900 rows read- which obviously then only writes 5,900 to my sink (SQL server in my instance)

NOTE: I'm also downloading companyProperties (79K rows) and contactProperties (470K rows). These are downloading fine and are not capping at 5,900 records

Is there something specifically that needs to be set in ADF or HubSpot to allow more than 5,900 contacts to be downloaded?

Vinodh247 13,626 Reputation points

2024-07-25T00:01:44.7133333+00:00

HubSpot enforces API rate limits that might affect your data retrieval. Make sure you are not hitting these limits. Monitor your API usage in HubSpot and consider implementing retries with appropriate delays if necessary.
PRADEEPCHEEKATLA-MSFT 85,586 Reputation points Microsoft Employee

2024-07-25T03:21:31.8766667+00:00
Grant Barrington - Thanks for the question and using MS Q&A platform.

There is no specific limitation of the HubSpot Source connector for Contact records to 5,900 records. However, there could be several reasons why you are only getting 5,900 records:

HubSpot API Limitations: HubSpot has API limitations that restrict the number of API calls and the amount of data that can be retrieved in a single API call. It's possible that the HubSpot Source connector is hitting one of these limitations, which is causing it to only retrieve 5,900 records.

Azure Data Factory Limitations: Azure Data Factory has its own limitations on the amount of data that can be processed in a single pipeline run. It's possible that the pipeline is hitting one of these limitations, which is causing it to only retrieve 5,900 records.

HubSpot Source Connector Configuration: It's possible that the HubSpot Source connector is not configured correctly, which is causing it to only retrieve 5,900 records. You may want to review the connector configuration and ensure that it's set up correctly.

To troubleshoot this issue, you can try the following:

Check the HubSpot API limits: You can check the HubSpot API limits to see if the connector is hitting any of these limits. You can also try to increase the API limits if necessary.

Check the Azure Data Factory limits: You can check the Azure Data Factory limits to see if the pipeline is hitting any of these limits. You can also try to increase the limits if necessary.

Review the HubSpot Source Connector Configuration: You can review the HubSpot Source connector configuration to ensure that it's set up correctly. You may want to check the connector documentation for any specific configuration requirements.

Contact HubSpot Support: If you are still unable to resolve the issue, you may want to contact HubSpot support for further assistance. They may be able to provide additional insights into the issue.

Hope this helps. Do let us know if you any further queries.
Grant Barrington 0 Reputation points

2024-07-25T23:28:58.5533333+00:00

@Vinodh247 I've checked the App in HubSpot and looked at the API calls, and all responses are 200's. There are no 429 errors (https://developers.hubspot.com/docs/api/usage-details#rate-limits)

Share via

Is there a limitation of the HubSpot Source connector for Contact records to 5,900 records?