Azure Synapse Data Factory REST Connector issues within Data Flows

Rob Richardson 1

We're trying to use the REST connector as a Source within a Data Flow which is supposedly supported based on this Announcement. However, when we attempt to Import the Schema for the endpoint, the process never completes; just seeing a loading indicator for an indefinite amount of time until the Debug Session eventually times out. The same endpoint works just fine when using it from an HTTP Connector. Is this a known issue?

Saurabh Sharma 23,676 Reputation points Microsoft Employee

2021-09-11T00:17:40.583+00:00

Hi @Rob Richardson ,

Thanks for using Microsoft Q&A !!
I have tried using REST source for a public REST API from here and I could see projection coming fine in DataFlow.

Even Data Preview shows data correctly.

Is REST API internal to you ? Also, what is the format (JSON or XML) your API is returning? Have you tried changing the format under Source Options

Also, could you provide a sample data ?

Thanks
Saurabh
Saurabh Sharma 23,676 Reputation points Microsoft Employee

2021-09-13T20:53:14.457+00:00

Hi @Rob Richardson ,

Following up to know if you are still having issues.

Thanks
Saurabh
Rob Richardson 1 Reputation point

2021-09-13T23:58:08.25+00:00

Saurabh,
Thanks for your reply. This is a public API we maintain. It is protected with Basic Authentication & the format is JSON. This same API works just fine when using the HTTP Connector. This API also works fine when using the REST connector from a Copy Data Activity - I can succesfully import the schema and copy all data to a JSON file in Storage easily. The problem only arises when we attempt to use the REST connector as a Data Flow Source.

Also, we tried the public REST API that you mentioned above from within a Data Flow and it worked fine so I'm not sure what's going on - perhaps the issue is related to Basic Authentication? Regardless, the problem appears to be isolated to when the REST connector is used as a Source within a Data Flow.
Saurabh Sharma 23,676 Reputation points Microsoft Employee

2021-09-17T23:08:09.283+00:00

@Rob Richardson It may be an authentication issue. Can you please try running your dataflow and provide me the runId so that I can try checking in backend for issues.

Thanks
Saurabh
Saurabh Sharma 23,676 Reputation points Microsoft Employee

2021-09-24T18:52:30.537+00:00

Hi @Rob Richardson ,

We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Thanks
Saurabh
Rob Richardson 1 Reputation point

2021-09-24T19:28:22.693+00:00

Saurabh,
Here's a Run ID from a Debug Pipeline Execution: 1d3b29ca-e78d-4103-a3ac-144240cdd31d. At the time of writing, this pipeline has not completed. Furthermore, the data flow from this pipeline was never completed/validated because I'm still unable to get the "Import Projection" feature to work/finish for the Data Flow Source. As you can see from the screenshot provided, I've asked Synapse/Data Factory to Import the Projection/Schema of the Source and yet after 11 mins, it's still running. I've previously seen the same behavior continue running for 30+ mins. Without the Source projection/schema in place, I can't complete & validate the data flow prior to Pipeline execution.
Please advise.
Rob
Saurabh Sharma 23,676 Reputation points Microsoft Employee

2021-10-04T22:15:27.37+00:00

Hi @Rob Richardson ,
I tried to check internally but nothing much could be concluded the information I could see. Can you please open a Support ticket for this issue so that an engineer can look into this issue. In case you have limitation creating a support please let me know and I will help you providing one-time free support ticket.

Thanks
Saurabh