ADF - Pagination with Headers not working correctly

Question

ADF - Pagination with Headers not working correctly

Aleix 5

Hello, I'm trying to list all our users by using the following API [1]. I already created the REST linked service and when testing by creating a dataflow and clicking on "Data preview", I can see some rows. The API response has a limit of 200 and we have thousands of users, so we have to work with Pagination. The next page is located in the response's headers, it doesn't appear in the response's body. This would be an example of what I see in Postman <https://xxx-xxx.okta.com/api/v1/users?after=12asd34&limit=200>; rel="next" This is my current configuration for the Source Activity inside my dataflow. Screenshot 2024-01-31 at 12.00.10 Then, when I preview the data, I receive the following error: Job aborted due to stage failure: Task 0 in stage 246.0 failed 1 times, most recent failure: Lost task 0.0 in stage 246.0 (TID 189) (vm-82869784 executor 1): com.microsoft.dataflow.Issues: DF-Rest_015 - Failure to read most recent page request: Illegal character in scheme name at index 0: <https://xxx-xxx.okta.com/api/v1/users?limit=200>; rel="self" If I remove the Pagination rule and preview the data again, when I see the key link , https://xxx-xxx.okta.com/api/v1/users?limit=200> is not there. Screenshot 2024-01-31 at 12.05.33

It looks like that ADF detects an invalid character, but the same content is what I receive through Postman. How can I fix this "invalid character" error message?

What configuration should I use for ADK to get the next page and show the data?

[1]

https://developer.okta.com/docs/reference/api/users/#list-users

Aleix 5 Reputation points

2024-02-01T09:28:05.87+00:00
Hello Satish, Thanks for replying! Please, see my responses below.

Provide a sample of Response JSON, how does it look like?

I’m attaching a JSON file (results.txt), where you can see an example of the response of /api/v1/users. Note all data has been redacted for security purposes. I’m also attaching a screenshot (screenshot.png) of the response’s headers. Interestingly, Okta returns 2 Keys for link (the parameter limit can change depending on the value you specify in the request):

<https://XXXXX/api/v1/users?limit=1>; rel="self"

<https://XXXXX/api/v1/users?after=67890&limit=1>; rel="next"

However, as explained in my first comment, the “Data Preview” for Azure ADF only shows 1 link.

Provide screenshot of the Dataflow "Data Preview" , we want to see how is the data preview?

I’m attaching another screenshot (preview.png)

Add screenshots of the postman when you want to compare the scenarios!!!

See screenshot.png

I just found this question in Okta's forum [1]. Apparently, other people faced exactly the same. Given that Okta provides 2 different lines of link, pagination does not seem to work correctly. Cheers, Aleix.

[1] https://devforum.okta.com/t/azure-data-factory-pagination-rfc5988/22453

preview.png screenshot.png result.txt
QuantumCache 20,676 Reputation points Moderator

2024-02-01T18:21:30.74+00:00

@Aleix Thanks for taking time and providing all that info, I appreciate it. We are looking into it!
Aleix 5 Reputation points

2024-02-05T07:07:03.46+00:00

Hey Satish, any news regarding this?
QuantumCache 20,676 Reputation points Moderator

2024-02-06T03:40:18.24+00:00

@Aleix Did you try the pagination with any other API service ither than Okta's?
Aleix 5 Reputation points

2024-02-06T07:25:29.9833333+00:00

Hello Satish, No, we have not used any other API server other than Okta, there isn't any other API we want to extract data through ADF. What I can tell you is that I contacted Okta’s Support and there isn’t any way for them to change or exclude 1 of the 2 “link” Keys. Cheers, Aleix.
QuantumCache 20,676 Reputation points Moderator

2024-02-06T17:49:26.4+00:00

@Aleix Thanks for the info, i am working on this, please stay tuned.
Aleix 5 Reputation points

2024-02-12T07:23:33.66+00:00

Hello Satish, Is there any news regarding this?
QuantumCache 20,676 Reputation points Moderator

2024-02-12T08:32:01.73+00:00

@Aleix Could you please confirm if the same Okta API is working from Postman? If the same API is working outside of ADF, then there should not be an issue from ADF side. I did check ADF pagination, and it is working as expected with pagination. I will check back on this, please stay tuned! When tested the 2 Links in the Response Header...! I see that only one Link is read in the Response Header. It may be too soon for me to comment that ADF can read multiple links in Response, So let me check further on this.
QuantumCache 20,676 Reputation points Moderator

2024-02-12T23:38:56.45+00:00

Aleix If the Response Header is returning the 2 Links with unique name, then I can see the ADF capture it
Aleix 5 Reputation points

2024-02-14T07:41:52.56+00:00

Hey Satish, My apologies for the delay. I just wanted to let you know that I created a support ticket to discuss this issue further. I think we can close this Q&A topic.
QuantumCache 20,676 Reputation points Moderator

2024-02-14T09:21:59.5766667+00:00

@Aleix Thank you for the response, please do share your findings in this regard, so that others will be aware of the solution, thanks for your understanding.

1 answer

Your answer

Aleix 5 Reputation points

2024-02-01T09:28:05.87+00:00

Hello Satish, Thanks for replying! Please, see my responses below.

Provide a sample of Response JSON, how does it look like?

I’m attaching a JSON file (results.txt), where you can see an example of the response of /api/v1/users. Note all data has been redacted for security purposes. I’m also attaching a screenshot (screenshot.png) of the response’s headers. Interestingly, Okta returns 2 Keys for link (the parameter limit can change depending on the value you specify in the request):

<https://XXXXX/api/v1/users?limit=1>; rel="self"

<https://XXXXX/api/v1/users?after=67890&limit=1>; rel="next"

However, as explained in my first comment, the “Data Preview” for Azure ADF only shows 1 link.

Provide screenshot of the Dataflow "Data Preview" , we want to see how is the data preview?

I’m attaching another screenshot (preview.png)

Add screenshots of the postman when you want to compare the scenarios!!!

See screenshot.png

I just found this question in Okta's forum [1]. Apparently, other people faced exactly the same. Given that Okta provides 2 different lines of link, pagination does not seem to work correctly. Cheers, Aleix.

[1] https://devforum.okta.com/t/azure-data-factory-pagination-rfc5988/22453

preview.png screenshot.png result.txt
QuantumCache 20,676 Reputation points Moderator

2024-02-01T18:21:30.74+00:00

@Aleix Thanks for taking time and providing all that info, I appreciate it. We are looking into it!
Aleix 5 Reputation points

2024-02-05T07:07:03.46+00:00

Hey Satish, any news regarding this?
QuantumCache 20,676 Reputation points Moderator

2024-02-06T03:40:18.24+00:00

@Aleix Did you try the pagination with any other API service ither than Okta's?
Aleix 5 Reputation points

2024-02-06T07:25:29.9833333+00:00

Hello Satish, No, we have not used any other API server other than Okta, there isn't any other API we want to extract data through ADF. What I can tell you is that I contacted Okta’s Support and there isn’t any way for them to change or exclude 1 of the 2 “link” Keys. Cheers, Aleix.
QuantumCache 20,676 Reputation points Moderator

2024-02-06T17:49:26.4+00:00

@Aleix Thanks for the info, i am working on this, please stay tuned.
Aleix 5 Reputation points

2024-02-12T07:23:33.66+00:00

Hello Satish, Is there any news regarding this?
QuantumCache 20,676 Reputation points Moderator

2024-02-12T08:32:01.73+00:00

@Aleix Could you please confirm if the same Okta API is working from Postman? If the same API is working outside of ADF, then there should not be an issue from ADF side. I did check ADF pagination, and it is working as expected with pagination. I will check back on this, please stay tuned! When tested the 2 Links in the Response Header...! I see that only one Link is read in the Response Header. It may be too soon for me to comment that ADF can read multiple links in Response, So let me check further on this.
QuantumCache 20,676 Reputation points Moderator

2024-02-12T23:38:56.45+00:00

Aleix If the Response Header is returning the 2 Links with unique name, then I can see the ADF capture it
Aleix 5 Reputation points

2024-02-14T07:41:52.56+00:00

Hey Satish, My apologies for the delay. I just wanted to let you know that I created a support ticket to discuss this issue further. I think we can close this Q&A topic.
QuantumCache 20,676 Reputation points Moderator

2024-02-14T09:21:59.5766667+00:00

@Aleix Thank you for the response, please do share your findings in this regard, so that others will be aware of the solution, thanks for your understanding.

Answer 1

@Aleix Looks like we need more information from your side to help troubleshoot your case!

Please provide a sample of Response JSON, how does it look like?
Please provide screenshot of the Dataflow "Data Preview" , we want to see how is the data preview?
Please add screenshots of the postman when you want to compare the scenarios!!!

Could you please confirm if the same Okta API is working from Postman? If the same API is working outside of ADF, then there should not be an issue from ADF side. I did check ADF pagination, and it is working as expected with pagination. I will check back on this, please stay tuned! When tested the 2 Links in the Response Header...! I see that only one Link is read in the Response Header. It may be too soon for me to comment that ADF can read multiple links in Response, So let me check further on this. User's image

User's image

If the Response Header is returning the 2 Links with unique name, then I can see the ADF capture it User's image

Share via

ADF - Pagination with Headers not working correctly

1 answer

Your answer