Access to API request logs from ADF Quickbooks Online pipeline?

Question

We are using the Azure Data Factory copy activity tool for Quickbooks Online https://learn.microsoft.com/en-us/azure/data-factory/connector-quickbooks?tabs=data-factory

We are getting intermittent errors that are hard to diagnose. We want to log the API requests that the copy activity tool is making to QBO to diagnose.

We followed the set up instructions here: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-log?tabs=data-factory

However, the log files in the storage account are only the headers, even when there is an error.

Example:

This was this error, from ADF activity run:

ErrorCode=UserErrorFailedToConnectOdbcSource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400
ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400
ERROR [HY000] [Microsoft][QuickBooks] (61) API Connection Failed. Bad Request. HTTP Response code: 400,Source=,'

The activity has confingured as:

Enable logging: true
Storage connection name: Storage_account
Logging level: Info
Logging mode: Best effort

The log files are found in my storage account but the content for all the log files is just: Timestamp,Level,OperationName,OperationItem,Message

Answer 1

Hi Ryan,

The error code 400 "bad request" means that you are formatting the API call wrong.

First I would try sending an API call with the following JSON.

https://learn.microsoft.com/en-us/azure/data-factory/connector-quickbooks?tabs=data-factory#linked-service-properties

{
    "name": "QuickBooksLinkedService",
    "properties": {
        "type": "QuickBooks",
        "typeProperties": {
            "connectionProperties": {
                "endpoint": "quickbooks.api.intuit.com",
                "companyId": "<company id>",
                "consumerKey": "<consumer key>", 
                "consumerSecret": {
                     "type": "SecureString",
                     "value": "<clientSecret>"
            	},
                "refreshToken": {
                     "type": "SecureString",
                     "value": "<refresh token>"
            	},
                "useEncryptedEndpoints": true
            }
        }
    }
}

For assistance with creating pipelines and activities in ADF I would reference:

https://learn.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities?tabs=data-factory

If this is helpful please accept answer.

Answer 2

Hi @Ryan Knight

For QuickBooks Online connector in Azure Data Factory is currently in preview, and there may be limitations or changes to the logging and monitoring capabilities of the connector.

In this case, I recommend checking the Azure Data Factory documentation and the QuickBooks Online API documentation to see if there are any known issues or limitations with the connector or the API that could be affecting the logging and monitoring capabilities.

If there are no known issues or limitations, and you are still unable to log the error details in detail, you may need to consider using a different tool or method to log the API requests and responses, such as a network traffic capture tool like Fiddler or Wireshark(not free tools).

You can also try reaching out to Azure support or QuickBooks support for assistance with diagnosing the issue and enabling more detailed logging and monitoring.

I hope this helps. Let me know if you have any further questions or concerns.