This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
We have a 500.000 image labeling project in progress. The team was receiving some errors that indicated no work was queued even though the project was only 3% complete.
We paused the project and attempted to restart to see if that would clear out the issue but when trying to restart an error is being thrown "Failed" with no additional information.
The issue has been resolved. There was some corruption in the database that was corrected by the product team.
@Dave Hartman Thanks for the question. Please share details of your experiment and issue from the ml.azure.com portal for a service engineer to lookup the issue from the back-end? This option is available from the top right hand corner of the portal by clicking the smiley face, Please select the option Microsoft can email you about the feedback along with a screen shot so our service team can lookup and advise through email.
My labeling project is also now frozen after an attempt to resume from paused... it has been trying to resume for >2 hours now....
After about 3-4 hours, I receieved a notification in the portal stating 'Project Failed' and the labling project went back to the 'paused' state'. I attempted to start the labeling project and it is now 'frozen' again.
Thanks in Advance,
eem
As mentioned in ramr-msft 's reply, please use the smiley face in the UI to provide feedback or report an issue, so that we know how to reach out and and provide update on your issue.
Besides a screenshot, if you can provide the project ID, that will be really helpful. The project ID is a GUID string showing in your URL, like "b025d4ff-3205-6144-19f6-fdad88269c3f" in the case of "https://ml.azure.com/labeling/project/b025d4ff-3205-6144-19f6-fdad88269c3f/...".
Thanks!
Hello,
I submitted the 'Feedback' form with a screenshot not long ago... I can submit another. I will include the project id in the form... thanks... the project is currently in 'failed/paused' state. I think it is possibly related to a frozen inference job state record in backend... or some other ML assist RI issue...
Hello Shu,
I responded as well through e-mail...
Hello,
After the 'backend' fix to this project, I was able to resume and label now by also turning off the ML Assist which appeared to be the inference job was holding the lock and was stuck. A warning icon appeared on the project's lower section listing ML Assist jobs and stated the inference job could no longer find it's compute. Disabling ML Assist had no impact on being able to resume prior to the backend change as the project would remain in the paused state after attempting to resume for a couple of hours...assuming this was some sort of time out.. anyway.. The project is now running again.
On a related note...You (or anyone) wouldn't happen to have a pointer to or access to C# code using ML.NET to pre-process and score images and then render the mask points on the results using when using the Mask rCNN models generated from the AutoML job processing the label data would you please? The fastcnn ML.NET code is being generated by Model Builder in VS2022, but I can't seem to find C# code with ML.NET for the MaskCNN model being generated by AutoML jobs from this 'mask' oriented label project.
I am able to use the fastcnn generated ONNX models from the labeled data no issues with ML.NET and .NET6...
Thanks Again!
eem
Hello Again.... now another label project in the same workspace with ML Assist 'Training' job is failing at the very end when attempting to start the Inferencing job...
Encountered an internal AutoML error. Error Message/Code: UserErrorException. Additional Info: UserErrorException:
Message: Request failed (400): Invalid dataflow specified. Execution failed with exception. InternalServerError,
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Request failed (400): Invalid dataflow specified. Execution failed with exception. InternalServerError, "
}
}
The training job appears to run ok as the output is produced into the blobstore correctly.
Thanks,
eem