Lookup Activity - Support REST Dataset in Azure Data Factory

Question

Hi,
I need to lookup a Rest API using OAuth2 client credentials to get a download link for a file. Per now this is not supported in the Lookup activity and OAuth2 is unsupported in data flow due to the following bug.
https://learn.microsoft.com/en-us/answers/questions/938311/azure-data-factory-data-flow-rest-api-access-token.html

Do you have a suggested workaround for this? If yes, can you please post it?

And, is there a timeline for the Rest API support in Lookup? I see this feature was suggested three years ago here. https://feedback.azure.com/d365community/idea/49dfea4c-7026-ec11-b6e6-000d3a4f032c

Br,
Audun

Answer 1

Hello @Audun Lie Indergaard and welcome to Microsoft Q&A.

As I understand, the goal is to do the equivalent of a Lookup on a Rest API. A Lookup is different from a Copy in that the results are made available to pipeline expressions. A Web activity also makes the results available to the pipeline expressions, however additional parsing may be needed.

There is another option to try. Use Lookup activity with HTTP instead of REST. One major difference between HTTP and REST, is the lack of Pagination support. However, given the limits on Lookup (Max 5000 rows or 4MB), pagination would not help. The other difference is Authentication options. Given OAuth isn't part of the HTTP Linked service, you would need to set the Linked Service Auth to Anonymous, and get the token in separate Web activity. Pass the token from Web activity output to Lookup HTTP activity, where it either is passed down to the Linked Service Auth headers, or used in the Additional Headers in Lookup Activity. Need to test which works better.

Are you blocked in Web activity by the same error as in the link you shared? If not, I can help with the parsing.
If you are blocked, we can try doing a Copy, and then do Lookup on the Copy sink.

I can ask the devs about the roadmap, but I'm not sure I will get an answer.