Dataflow runs indefinitely - no errors & no output

Question

Hi,

I have a big dataflow (100+ activities) which has lot of memory intensive operations like lookups & joins. Normally the dataflow runs under 256 (+ 16 driver cores) Memory optimized configuration only however, the input data row count is very low only 16000. The dataflow runs indefinitely no errors no output nothing it simply keeps running.

In debug mode I picked only 500 rows and few activities ran properly as expected however, few activities did not yield any output buy simply say 'Timeout error'. I don't understand why it keeps running. It is hard to debug without proper error info too. Timeout error is not very informative. Could someone please help

Regards,
_arc

Answer 1

Hello @_arc ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is to find out as to what is the source of time out error , please do let us know if its not accurate.
You never mentioned if the pipeline has every worked fine .But since you mentioned that you have 100 + activitites , I think I will try and use the custom compute size and see if that helps

Also while debugging I will set the Time to live to max of 4 hrs .

Also while the debug session is "On" you can move around the activities from left to right
and select the activity and check the "refresh" this may be able to give the info as to whcih activities are fetching data , but keeping in mind that you have so many activities it may be diffcult .

Please do let me if you have any queries.
Thanks
Himanshu

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators