This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 10%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
I saw the recent release of the Dataflow "Quick re-use" feature and immediately enabled it (the start-up time of pre-warmed clusters had been something that had been an issue for us for age)
Unfortunately, I think this feature is buggy.
Ever since enabling this setting, I've been intermittently having activity spontaneously hang and then error with:
"The databricks job of Dataflow completed, but the runtime state is either null or still InProgress."
It's happened on a variety of different dataflows, with nothing in common other than "they all use the same IR, which now has 'Quick re-use' enabled".
In case it's of use, one such run would be:
Pipeline Run Id: 0198122b-6bb1-4250-9fcc-3d28417366e7
Activity Run Id: 1141af06-376a-4554-97da-c36a79278113
"dataFlowETag": "0e006849-0000-0c00-0000-6065e9da0000"
But I've had numerous other instances of it happening.
An Azure service for ingesting, preparing, and transforming data at scale.
More examples:
0198122b-6bb1-4250-9fcc-3d28417366e7
1141af06-376a-4554-97da-c36a79278113; dataFlowETag: 0e006849-0000-0c00-0000-6065e9da0000 (Above) fed793a2-595e-4578-a230-fe118684015e; dataFlowETag: 0e006249-0000-0c00-0000-6065e9da0000 1d69b761-6cf2-4a70-96a9-555184f2db09
17f058e0-51bb-4739-ad65-2ba751dda134; dataFlowETag: 3c000239-0000-0c00-0000-608819be0000 b497dd6e-b55f-47e3-a131-813ecdb9d9d9; dataFlowETag: 3c001b39-0000-0c00-0000-608819be0000 ![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Hi @Mike McLean (Shell) ,
Thanks for using Microsoft Q&A !!
I am checking internally with the products team on the same and will update you back on the same. Thanks for sharing your run-id's as well and this would be helpful. I will let you know in case I will need anything else from you.
Thanks
Saurabh
Hi @Mike McLean (Shell) ,
We haven't heard back from you. Just wanted to check if you are you still facing the issue? Please send us an email if you need help on creating a support ticket.
Thanks
Saurabh
@Saurabh Sharma
You didn't ask for any more information; what were you expecting to hear back from me about? You said you were going to update me!
Have you released a fix that you think would solve the bug?
Yes, as far as I know it's still broken?
You didn't ask me to raise a support ticket, as far as I know these forums ARE the way you report bugs to Microsoft.
@Mike McLean (Shell) I am extremely sorry for the confusion as the above reply was meant for some other thread.
To update you on the latest - this definitely is an issue which the products team is working now to provide a fix in the production environment (probably by this week). I am tracking the same internally and update you as soon as it gets deployed in the production environment.
Thanks
Saurabh
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP5%3C/text%3E%3C/svg%3E)
Hi @Saurabh Sharma ,
We have seen this issue over the weekend after switching over to Quick re-use; prior to that, pipeline had been running for months with no issue. This happened only one of the three executions since.
Pipeline run id: d3883e94-bad2-42a7-94be-aa2c9b351227
Activity run id: 3ec612bd-a499-4fe4-86b4-28d2ae5a4461
Integration runtime is in Australia East.
Can you please let us know what our expectations should be on this based on defect/fix/region status?
Thanks,
Patrick
Hi @Saurabh Sharma ,
Any update on this? I am still seeing this issue and it's affecting our production loads.
If there is a better forum to post on , or I if I need to open a new thread with this same issue, I will do so., but we need to get this sorted.
Thanks,
Patrick
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Hi,
We are facing the same issue on our subscription with adf resources in North Europe today the 8th of July 2021.
Are you deploying a new version or patch ?
Please Help!
Thanks!
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 57.99999999999999%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENF%3C/text%3E%3C/svg%3E)
Hello,
I have the very same issue, I'm waiting for a situation update
We could execute our pipelines by changing the location of the runtime to west europe...temporarily
Hi @Mike McLean (Shell) ,
The fix has been deployed in production for North Europe region. Please check and let me know if you see any issues.
Thanks
Saurabh
Excellent news. I'll watch for any further re-ocurrences.
Alas it wasn't a particularly reliable error, so I can't explicitly test it, but I will monitor for a week and update you with the results.
We've monitored for a week, and have seen no further issues. It does seem to have been resolved. Many thanks!
@Mike McLean (Shell) Great. Thanks for the confirmation.