Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,373 questions
Azure Synapse Data Factory REST Connector issues within Data Flows
Rob Richardson
1
Reputation point
We're trying to use the REST connector as a Source within a Data Flow which is supposedly supported based on this Announcement. However, when we attempt to Import the Schema for the endpoint, the process never completes; just seeing a loading indicator for an indefinite amount of time until the Debug Session eventually times out. The same endpoint works just fine when using it from an HTTP Connector. Is this a known issue?
Azure Synapse Analytics
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,623 questions
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator2021-09-11T00:17:40.583+00:00 Hi @Rob Richardson ,
Thanks for using Microsoft Q&A !!
I have tried using REST source for a public REST API from here and I could see projection coming fine in DataFlow.
Even Data Preview shows data correctly.
Is REST API internal to you ? Also, what is the format (JSON or XML) your API is returning? Have you tried changing the format under Source Options
Also, could you provide a sample data ?
Thanks
Saurabh -
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator
2021-09-13T20:53:14.457+00:00 -
Rob Richardson 1 Reputation point
2021-09-13T23:58:08.25+00:00 Saurabh,
Thanks for your reply. This is a public API we maintain. It is protected with Basic Authentication & the format is JSON. This same API works just fine when using the HTTP Connector. This API also works fine when using the REST connector from a Copy Data Activity - I can succesfully import the schema and copy all data to a JSON file in Storage easily. The problem only arises when we attempt to use the REST connector as a Data Flow Source.Also, we tried the public REST API that you mentioned above from within a Data Flow and it worked fine so I'm not sure what's going on - perhaps the issue is related to Basic Authentication? Regardless, the problem appears to be isolated to when the REST connector is used as a Source within a Data Flow.
-
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator
2021-09-17T23:08:09.283+00:00 @Rob Richardson It may be an authentication issue. Can you please try running your dataflow and provide me the runId so that I can try checking in backend for issues.
Thanks
Saurabh -
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator
2021-09-24T18:52:30.537+00:00 Hi @Rob Richardson ,
We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.
Thanks
Saurabh -
Rob Richardson 1 Reputation point
2021-09-24T19:28:22.693+00:00 Saurabh,
Here's a Run ID from a Debug Pipeline Execution: 1d3b29ca-e78d-4103-a3ac-144240cdd31d. At the time of writing, this pipeline has not completed. Furthermore, the data flow from this pipeline was never completed/validated because I'm still unable to get the "Import Projection" feature to work/finish for the Data Flow Source. As you can see from the screenshot provided, I've asked Synapse/Data Factory to Import the Projection/Schema of the Source and yet after 11 mins, it's still running. I've previously seen the same behavior continue running for 30+ mins. Without the Source projection/schema in place, I can't complete & validate the data flow prior to Pipeline execution.
Please advise.
Rob
-
Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator
2021-10-04T22:15:27.37+00:00 Hi @Rob Richardson ,
I tried to check internally but nothing much could be concluded the information I could see. Can you please open a Support ticket for this issue so that an engineer can look into this issue. In case you have limitation creating a support please let me know and I will help you providing one-time free support ticket.Thanks
Saurabh
Sign in to comment
Sign in to answer