Azure Data Factory: Web Activity vs HTTP Linked Service: Web can get response from an API, HTTP cannot!

Question

Azure Data Factory: Web Activity vs HTTP Linked Service: Web can get response from an API, HTTP cannot!

Sam Lohbeck 0

I am currently struggling with a persistent issue in Data Factory.

In the simplest terms, I have an endpoint I'm trying to reach to get a JSON response. Let's call it *"*https://myapi.company.com/attendeeinfo.json**"

I have no authentication headers or anything, the ApiToken and the credentials are being sent inside the POST Body.

When I create a Web Activity (picture for clarity), I plug in all the info, and it works great:
User's image

But when I create an HTTP Dataset and a HTTP Linked Service, (or REST) it breaks-down immediately.

Here is my Linked Service (it successfully test-connects):

User's image

And here is my DataSet:
User's image

Now, when I go to Preview Data, or Import Schema, or any sort of preview feature I get a 400 BadRequest error:
User's image

User's image

Taking a BadRequest for what it is we can assume my JSON formatting is wrong, or something is misspelled or something of the likes of that (I've also tried making the baseUrl end with '/' like it says). But when I copy and paste my body and full URL and plug it into Postman, it works perfectly fine, and I get a response.

I am at a complete loss. Has anybody experienced this before? Am I missing something? Any help would be appreciated!**

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-13T09:43:41.6+00:00
@Sam Lohbeck

Thank you for posting query in Microsoft Q&A Platform.

I understand that you're facing an issue with Data Factory while trying to reach an endpoint and retrieve a JSON response. You've mentioned that you've successfully tested the connection using a Web Activity, but encountering a 400 BadRequest error when using an HTTP Dataset and Linked Service.

Here are some potential reasons for the error and troubleshooting steps you can try:

Check the Input and Output Schema: Ensure that the schema of your mapping data flow activity matches the source and sink data stores.

Malformed Request Syntax or Invalid Parameters: The 400 status code means the server judged the client’s request to be invalid in some way. This could be due to malformed request syntax, missing required headers, parameters or request body, or invalid parameter values passed in the request.
refer :https://devhunt.org/blog/400-bad-request-in-rest-apis-how-to-handle-and-prevent

Trailing Whitespace in Pipeline Execution Tasks: One user found that the name of one of their Pipeline execution tasks had a trailing whitespace. Once this was removed, the master pipeline was able to run successfully.

Missing Comma in @concat() Dynamic Expression: issue might due to a missing comma in a @concat() dynamic expression.

1. Mismatched Request Body Format:

Double-check that the request body format in your Data Factory matches the format expected by the API. While Postman might handle different formats gracefully, Data Factory might be stricter. Ensure it's sent as JSON as you mentioned.

Use the built-in Data Factory data flow features to format and validate your JSON data before sending it to the API.

2. Authentication Issues:

Even though you mentioned no explicit authentication headers, some APIs might require other forms of authentication like Basic Auth encoded in the URL or API key in a specific header.

Double-check the API documentation for any authentication requirements and configure them accordingly in your Linked Service settings.

3. Firewall or Network Restrictions:

There might be firewall restrictions or network policies in your Azure environment that are blocking the communication between Data Factory and the API endpoint.

Verify with your network administrator if there are any restrictions in place and if necessary, request temporary access for testing purposes.

Hope this helps. Do let us know if you any further queries.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-15T18:21:44.86+00:00

@Sam Lohbeck
I understand, dealing with API providers can sometimes be challenging. However, in this case, it might be the most effective way to resolve the issue. They should be able to provide more specific details about the request they are receiving and why it might be resulting in a BadRequest error.

Before reaching out to them, you might want to gather as much information as possible about the request you’re sending. This includes the exact URL, headers, and body of the request. Also, make sure to note down any specific error messages you’re receiving. Remember, clear and concise communication can often help in getting a quicker and more accurate response.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-19T04:42:31.85+00:00

@Sam Lohbeck We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-15T18:21:44.86+00:00

@Sam Lohbeck
I understand, dealing with API providers can sometimes be challenging. However, in this case, it might be the most effective way to resolve the issue. They should be able to provide more specific details about the request they are receiving and why it might be resulting in a BadRequest error.

Before reaching out to them, you might want to gather as much information as possible about the request you’re sending. This includes the exact URL, headers, and body of the request. Also, make sure to note down any specific error messages you’re receiving. Remember, clear and concise communication can often help in getting a quicker and more accurate response.
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-02-19T04:42:31.85+00:00

@Sam Lohbeck We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Richard Swinbank 527 MVP

I'm guessing you're creating a dataset prior to configuring a pipeline Copy activity? For copying data the HTTP method (POST) and the POST body are specified in the Copy Activity itself.

Notice that the dataset has no option to specify a HTTP method or a body. This means that when you try to preview data in the dataset, ADF sends a request, probably GET, without a body. I expect this isn't valid for your web service, which is why you're getting the 400.

Create the Copy activity in the pipeline and configure its source to use your dataset. You'll then be able to set a request method & body and will be able to preview the dataset there.

Sam Lohbeck 0 Reputation points

2024-02-13T14:48:31.99+00:00

Thank you for the clarification!

I created a new Copy Activity and created a new Source Dataset from within it. I copy and pasted the Body from my successful Web activity. I received a 400 again here: ☹️
Richard Swinbank 527 Reputation points MVP

2024-02-13T15:39:19.19+00:00

~~Sorry to hear that Sam. The image you posted isn't showing up for some reason. Can you post another one that I can look at?~~ Ignore that, it's appeared!
Richard Swinbank 527 Reputation points MVP

2024-02-13T16:01:10.9366667+00:00

That looks OK. The fact that you're getting a 400 at all means that your request is getting to the API endpoint -- it's just being rejected. Are you able to talk to the provider of the API and see why it's being rejected at their end?

It would be great to be able to compare the HTTP requests submitted by the Web & Copy activities respectively, but unforunately that detail isn't exposed by ADF.
Sam Lohbeck 0 Reputation points

2024-02-13T16:49:54.97+00:00

That was a big investigation of mine was trying to figure out where I could compare the submitted requests. Good to know that we don't really have access to see that.

It is possible to talk to the API provider (they're just...very difficult to work with lol). I will start down that path if that is the best way to find out what's going on!

Share via

Azure Data Factory: Web Activity vs HTTP Linked Service: Web can get response from an API, HTTP cannot!

1 answer

Your answer