Failure happened on source side while extracting data from REST API

Question

Failure happened on source side while extracting data from REST API

Anuganti Suresh 200

I want to extract JSON file REST API. By using WEB activity with URL, POST method & Body generating access token. Later copy activity connected to web. Source side getting error while connecting REST API using Linked Service (Base URL, Anonymous, Authorization headers with "Bearer XXXXXXXXXXXXX".

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 404 NotFound, please check your activity settings.

Request URL: https://base.com/v1/xxxxxxxxxxx/activity_audit

Response: ,Source=Microsoft.DataTransfer.ClientLibrary,'

Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-12T12:35:27.75+00:00
Hi @Anuganti Suresh

Thank you for reaching out to us with your query.
Based on the error message you provided, it seems like there is an issue with the REST API call from the Web activity in Azure Data Factory. Here are some steps you can take to troubleshoot the issue:

Verify that the REST API URL is correct and accessible. You can try accessing the URL from a web browser or a REST client like Postman to see if it returns the expected response.

Check if the access token generated by the Web activity is valid and has the necessary permissions to access the REST API. You can check the Azure documentation on how to generate an access token for REST API authentication.

Ensure that the Linked Service configuration is correct. Check if the Base URL, Anonymous, and Authorization headers with "Bearer XXXXXXXXXXXXX" are correctly configured.

Check if the Copy activity is correctly configured to use the Linked Service. Ensure that the source and destination datasets are correctly configured.

If the above steps do not resolve the issue, you can try enabling debug mode in Azure Data Factory to get more detailed error messages. You can also check the Azure Data Factory documentation for more information on how to troubleshoot errors.

I hope this helps you resolve the issue. Let me know if you have any further questions or need more information.
Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-13T11:15:15.63+00:00

@Anuganti Suresh We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anuganti Suresh 200 Reputation points

2023-12-13T11:58:50.1466667+00:00

@Smaran Thoomu , still not resolved issue.
currently Linked Service created with Base URL & Anonymous. Authorization headers with "Bearer XXXXXXXXXXXXX" provided under copy activity (source). Authorization headers need to provide in Linked Service itself? if yes how to dynamically pass?
Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-14T13:14:15.67+00:00
Hi @Anuganti Suresh
Thank you for getting back to me. If you need to pass the Authorization header dynamically, you can use Azure Key Vault to store the access token and retrieve it at runtime in your pipeline. Here are the steps to do this:

Create an Azure Key Vault and store the access token as a secret in the Key Vault.

Create a Linked Service for the Key Vault in Azure Data Factory.

In your pipeline, add a Lookup activity to retrieve the access token from the Key Vault using the Linked Service.

Use a Set Variable activity to set the access token as a pipeline variable.

In your Web activity, use the pipeline variable to dynamically set the Authorization header with the access token.

Here's an example of how to use the Lookup activity to retrieve a secret from Azure Key Vault:

{ "name": "LookupAccessToken", "type": "Lookup", "dependsOn": [], "userProperties": [], "typeProperties": { "source": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "AzureKeyVaultLinkedService", "type": "LinkedServiceReference" }, "secretName": "AccessToken" }, "dataset": { "referenceName": "AccessToken", "type": "DatasetReference" } } }

In this example, the Lookup activity retrieves the secret named "AccessToken" from the Azure Key Vault using the Linked Service named "AzureKeyVaultLinkedService". The retrieved secret is stored in the dataset named "AccessToken". You can then use the Set Variable activity to set the value of the pipeline variable to the retrieved secret and use the pipeline variable in the Authorization header of the Web activity.

Your answer

Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-12T12:35:27.75+00:00

Hi @Anuganti Suresh

Thank you for reaching out to us with your query.
Based on the error message you provided, it seems like there is an issue with the REST API call from the Web activity in Azure Data Factory. Here are some steps you can take to troubleshoot the issue:

Verify that the REST API URL is correct and accessible. You can try accessing the URL from a web browser or a REST client like Postman to see if it returns the expected response.

Check if the access token generated by the Web activity is valid and has the necessary permissions to access the REST API. You can check the Azure documentation on how to generate an access token for REST API authentication.

Ensure that the Linked Service configuration is correct. Check if the Base URL, Anonymous, and Authorization headers with "Bearer XXXXXXXXXXXXX" are correctly configured.

Check if the Copy activity is correctly configured to use the Linked Service. Ensure that the source and destination datasets are correctly configured.

If the above steps do not resolve the issue, you can try enabling debug mode in Azure Data Factory to get more detailed error messages. You can also check the Azure Data Factory documentation for more information on how to troubleshoot errors.

I hope this helps you resolve the issue. Let me know if you have any further questions or need more information.
Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-13T11:15:15.63+00:00

@Anuganti Suresh We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Anuganti Suresh 200 Reputation points

2023-12-13T11:58:50.1466667+00:00

@Smaran Thoomu , still not resolved issue.
currently Linked Service created with Base URL & Anonymous. Authorization headers with "Bearer XXXXXXXXXXXXX" provided under copy activity (source). Authorization headers need to provide in Linked Service itself? if yes how to dynamically pass?
Smaran Thoomu 25,495 Reputation points Microsoft External Staff Moderator

2023-12-14T13:14:15.67+00:00

Hi @Anuganti Suresh
Thank you for getting back to me. If you need to pass the Authorization header dynamically, you can use Azure Key Vault to store the access token and retrieve it at runtime in your pipeline. Here are the steps to do this:

Create an Azure Key Vault and store the access token as a secret in the Key Vault.

Create a Linked Service for the Key Vault in Azure Data Factory.

In your pipeline, add a Lookup activity to retrieve the access token from the Key Vault using the Linked Service.

Use a Set Variable activity to set the access token as a pipeline variable.

In your Web activity, use the pipeline variable to dynamically set the Authorization header with the access token.

Here's an example of how to use the Lookup activity to retrieve a secret from Azure Key Vault:

{ "name": "LookupAccessToken", "type": "Lookup", "dependsOn": [], "userProperties": [], "typeProperties": { "source": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "AzureKeyVaultLinkedService", "type": "LinkedServiceReference" }, "secretName": "AccessToken" }, "dataset": { "referenceName": "AccessToken", "type": "DatasetReference" } } }

In this example, the Lookup activity retrieves the secret named "AccessToken" from the Azure Key Vault using the Linked Service named "AzureKeyVaultLinkedService". The retrieved secret is stored in the dataset named "AccessToken". You can then use the Set Variable activity to set the value of the pipeline variable to the retrieved secret and use the pipeline variable in the Authorization header of the Web activity.

Share via

Failure happened on source side while extracting data from REST API

Your answer