question

RobRichardson-9222 avatar image
0 Votes"
RobRichardson-9222 asked SaurabhSharma-msft commented

Azure Synapse Data Factory REST Connector issues within Data Flows

We're trying to use the REST connector as a Source within a Data Flow which is supposedly supported based on this Announcement. However, when we attempt to Import the Schema for the endpoint, the process never completes; just seeing a loading indicator for an indefinite amount of time until the Debug Session eventually times out. The same endpoint works just fine when using it from an HTTP Connector. Is this a known issue?


azure-data-factoryazure-synapse-analytics
· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @robrichardson-9222,

Thanks for using Microsoft Q&A !!
I have tried using REST source for a public REST API from here and I could see projection coming fine in DataFlow.
131202-image.png
Even Data Preview shows data correctly.
131224-image.png

Is REST API internal to you ? Also, what is the format (JSON or XML) your API is returning? Have you tried changing the format under Source Options
131232-image.png

Also, could you provide a sample data ?

Thanks
Saurabh


0 Votes 0 ·
image.png (24.0 KiB)
image.png (25.1 KiB)
image.png (32.2 KiB)

Hi @RobRichardson-9222,

Following up to know if you are still having issues.

Thanks
Saurabh

0 Votes 0 ·

Saurabh,
Thanks for your reply. This is a public API we maintain. It is protected with Basic Authentication & the format is JSON. This same API works just fine when using the HTTP Connector. This API also works fine when using the REST connector from a Copy Data Activity - I can succesfully import the schema and copy all data to a JSON file in Storage easily. The problem only arises when we attempt to use the REST connector as a Data Flow Source.

Also, we tried the public REST API that you mentioned above from within a Data Flow and it worked fine so I'm not sure what's going on - perhaps the issue is related to Basic Authentication? Regardless, the problem appears to be isolated to when the REST connector is used as a Source within a Data Flow.

0 Votes 0 ·

@robrichardson-9222 It may be an authentication issue. Can you please try running your dataflow and provide me the runId so that I can try checking in backend for issues.

Thanks
Saurabh

0 Votes 0 ·
Show more comments

Saurabh,
Here's a Run ID from a Debug Pipeline Execution: 1d3b29ca-e78d-4103-a3ac-144240cdd31d. At the time of writing, this pipeline has not completed. Furthermore, the data flow from this pipeline was never completed/validated because I'm still unable to get the "Import Projection" feature to work/finish for the Data Flow Source. As you can see from the screenshot provided, I've asked Synapse/Data Factory to Import the Projection/Schema of the Source and yet after 11 mins, it's still running. I've previously seen the same behavior continue running for 30+ mins. Without the Source projection/schema in place, I can't complete & validate the data flow prior to Pipeline execution.
Please advise.
Rob
135095-image.png



0 Votes 0 ·
image.png (83.9 KiB)
Show more comments

0 Answers