dataflow failed in prod but sucessed in dev. for moving with json code

Question

Hi team

Dataflow : purpose converting Json into CSV flattern and push to data lake gen2 from jsoin filke whicj is there in datalake gen 2

when i tested in dev dataflow working fine, but unable to move in prod and test with json code movement since my adf dont have git.

i have tested dataflow in dev its working properly. i have recreated dataset for source and sink in prod and copy paste json code, after that i am getting error as shown in attachment. how can i solve as early as possible.

for dev i can preview data as well for sake of u r undestanding i have attached a screenshort.

kindly help to fix as soon as posible. its impacting huge.

Answer 1

In the first screenshot, error says that you have used Self hosted IR in linked services. Is that the case? Did you create these linked services by copying the JSON code from Dev Linked service? Can you paste the Json code behind both the linked services?

I guess, you need to remove the self hosted IR and use Azure IR instead, and test the linked service connection.

Answer 2

Hello @Karnati,Venkata Suchendra Reddy,IN-Bangalore ,

Thanks for the question and using MS Q&A platform.

My understanding is that you are receiving above validation errors, while using SHIR in your Mapping Data flow dataset linked service. Please correct if I'm wrong.

As called out by anonymous userChaudhari , currently Self Hosted Integration runtime is not supported in Mapping Data flows which is why you are receiving errors mentioned in your post.

To overcome this issue, you will need to replace the SHIR with Azure IR in your mapping data flows.

The strange thing I would like to reconfirm is, in your DEV Datafactory -

1. Could you please re-confirm if you are using a Dataset with SHIR in your Mapping dataflow activity?
2. If yes, would request you to share a screenshot of the data set with SHIR configured and the mapping dataflow in which that dataset (using SHIR) is being referenced/used? so that I can pass this information to respective product owners.

The reason would like clarification on this is because the public documentation clearly states that Data Flow activities are executed on their associated Azure integration runtime. The Spark compute utilized by Data Flows are determined by the data flow properties in your Azure IR and are fully managed by the service.

As called out by Vaibhav, the Integration Runtime documentation also calls out the same:

But to unblock, you will have to use Azure IR and there should be any issue in using Azure IR as it is the recommended IR to be used with Mapping data flows.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators