Data Factory Pipeline source pointing to Azure Data Lake failing

Question

Hello:

I have a Source in a Pipeline which points to a DataSet.

If I test the DataSet the connection tests fine and I can preview the data. The schema is also being picked up correctly. There is an extra attribute in the first position, called Prop_0. Do not know what that is about.

Now in my pipeline if I test the source "Test Connection" it fails.

In notifications I get BadRequest - RunId: 3d60c49d-b442-4c6f-bc56-831839088738

If I open the Dataset and test it, it tests fine.

Please note this Source worked earlier and I have a pipeline build based on the source.

Requesting your kind assistance.

KD

Answer 1

Hi @KBD

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

I have a Source in a Pipeline which points to a DataSet.

If I test the DataSet the connection tests fine and I can preview the data. The schema is also being picked up correctly. There is an extra attribute in the first position, called Prop_0. Do not know what that is about.

Now in my pipeline if I test the source "Test Connection" it fails.

In notifications I get BadRequest - RunId: 3d60c49d-b442-4c6f-bc56-831839088738

If I open the Dataset and test it, it tests fine.

Please note this Source worked earlier and I have a pipeline build based on the source.

Requesting your kind assistance.

Solution:

I refreshed my Azure Dev --> just clicked refresh and it started worked.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Hi @KBD

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

As I understand your issue you are facing an error: "BadRequest" when you are trying to test connection of a dataset inside your dataflow. Please let me know if that is not the case.

Make sure you are not using Self hosted IR in dataflow since Dataflow only supports Auto Resolve Integration Runtime

Point the linked service to AutoResolve Integration Runtime that is the default IR in Azure datafactory.

I tried to replicate the scenario with the help of JSON file as a dataset.

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.