While attempting to train a Neural Network , the following error occurs on Hyper-parameter tuning "Failed to query task; exceeded retry count for operation" after one minute only. The error message also occurs without an "error code".

Failed to query task; exceeded retry count for operation

Ezzeldin, Emad 16

While attempting to train a Neural Network , the following error occurs on Hyper-parameter tuning "Failed to query task; exceeded retry count for operation" after one minute only.
The error message also occurs without an "error code".

GiftA-MSFT 11,166 Reputation points

2021-07-20T17:21:48.997+00:00

Hi, thanks for reaching out. Are you able to run other experiments successfully?
Ezzeldin, Emad 16 Reputation points

2021-07-20T17:56:43.833+00:00

Thank you for replying.It is a common phenomenon among all my other Neural Network Experiments actually.
Ezzeldin, Emad 16 Reputation points

2021-07-20T19:36:00.943+00:00

"https://gallery.azure.ai/Experiment/Experiment-created-on-7-20-2021"

Here is the link to the experiment. As you open it, if you change the Number of iteration on the Neural Network to high ones say 1000,2000,4000 iterations, you will get the error mentioned above. While of course increasing iterations is very necessary to make the Neural Network fit on the data.
GiftA-MSFT 11,166 Reputation points

2021-07-20T20:03:24.657+00:00

Hi, thanks for sharing the experiment. I wasn't able to reproduce this issue. I tried increasing the number of iterations and it ran successfully. Can you share a screenshot of what the error looks like? I'm wondering if the issue has to do with your subscription or workspace.
Ezzeldin, Emad 16 Reputation points

2021-07-20T20:36:21.373+00:00

As you can see when you put 1000,2000,4000 iterations , it fails. It will first give me this error "does not support shuffling" and if I go to the NN and untick the shuffle examples button, it moves forward to the error I reported initially.
GiftA-MSFT 11,166 Reputation points

2021-07-20T21:49:25.167+00:00

Okay, I selected "random grid" as parameter sweeping mode that's why. It could be a limitation of ML classic. Will confirm and share updates soon.
Ezzeldin, Emad 16 Reputation points

2021-07-20T22:10:03.56+00:00

I selected Entire Grid and you can see the rest In this pic or in the gallery.
Ezzeldin, Emad 16 Reputation points

2021-07-20T22:55:58.35+00:00

Here is another run with Random Sweep. I turned Shuffling off on the NN. The error does not even has a code.
Ezzeldin, Emad 16 Reputation points

2021-07-20T23:02:03.223+00:00

This is how the error looks like as I hover over hypertuning.
WC 16 Reputation points

2021-07-21T23:36:17.007+00:00

I am also having this issue - on all my experiments, and on no specific module - it changes each time i try to run
WC 16 Reputation points

2021-07-21T23:43:28.467+00:00

Same issue in Microsoft Machine Learning Studio (classic) - it pops up on different modules each time so is not an issue with a specific module. Worked fine until a few days ago.

It occurs in all my experiments.
정현빈 1 Reputation point

2021-07-22T08:07:12.857+00:00

I am also having this issue
정현빈 1 Reputation point

2021-07-22T08:25:25.46+00:00

I also have a same problem.....
jorge parra 1 Reputation point

2021-07-22T20:59:16.233+00:00

Same problem
GiftA-MSFT 11,166 Reputation points

2021-07-22T22:11:55.293+00:00

Hi all, I have forwarded your feedback to the product team...will share updates as soon as possible. Thanks.
GiftA-MSFT 11,166 Reputation points

2021-07-28T21:46:32.417+00:00

Quick follow-up. A fix has been deployed to resolve this issue, please try again. If error persists, please let us know. Thanks!

Share via

Failed to query task; exceeded retry count for operation

Your answer