How to fix number of labels are not updated in modelconfig when running MMDetection model in Azure ML?

Clouddle 0

Hi teams, I am trying to train a mask2former model from mmdetection as per the offical notebook. All I did change was the model name. Importing model worked fine, but error 'Number of classes are not updated in modelconfig' raised. I believe this is system or environment error, and as a user I can do nothing about it. Frankly speaking, I have tried many models from mmdetection, all of them failed with various reasons, except the sample models used in official notebook. Even I import that model into my registry and try to fine-tune that model using pipeline, instead of using automl, it still failed with the same error. I really appreciate your work to develop this amazing platform, but this one could be the concern. Many thanks.

Clouddle 0 Reputation points

2024-02-01T10:24:01.31+00:00
Let me clarify a little,

Importing model worked fine, but error 'Number of classes are not updated in modelconfig' raised during fine-tune session.
YutongTie-MSFT 46,976 Reputation points

2024-02-02T01:33:05.6133333+00:00

@Clouddle Thanks for reaching out to us, could you please share the document you are referring to so that we can reproduce the issue and process? Regards, Yutong
Clouddle 0 Reputation points

2024-02-02T02:30:31.75+00:00
Hi, Yutong. Thank you for your reply. I tried this notebook, and I replaced the model name 'maskrcnn_resnet50_fpn' with 'mask2former_r50_8xb2-lsj-50e_coco' in .set_training_parameters method, as per the model name in mmdetection metafile.yaml. This will raise error 'number of labels are not updated in modelconfig', come along with error 'list index out of range'.

And I also tried to import this model into my registry, following this notebook. Basically correct, but I found an error in this notebook. In section 2.3,

convert_model_to_mlflow = convert_model_to_mlflow_component( model_path=mmdetection_import_model.outputs.output_dir, mlflow_flavor="mmlab_pyfunc", task_name=task_name, )

Seems like mlflow_flavor is not a part of this component. Instead, model_framework = "MMLab" works great for me.

After importing the model to my own registry, I followed this notebook to do the fine-tuning, and the same error raised.

As I said, I have tried many models from MMDetections. I would like to share all the errors I've got w.r.t each model.

mask-r-cnn-convnext: This model is not supported. I read the MMDetection documentation, convnext backbone needs mmpretrain installing. And the environment to run import pipeline (acft-mmdetection-image-gpu-version-18) does not have mmpretrain installed. That may be the case.

mask2former: number of labels are not updated in model config.

cascade-mask-rcnn: border_allowed should be set to be -1 to skip the anchor condition.

rtmdet: some tensors share memory, this will lead to duplicate memory on disk and potential difference when loading them again. But this one may from huggingface.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
YutongTie-MSFT 46,976 Reputation points

2024-02-22T05:11:47.1066667+00:00
@Clouddle Thanks for the information and sorry for the late reply, I am contacting the content team for above error mentioned, at the meantime, could you please check if below workaround is helpful?

Updating the number of labels in the model config: The error message suggests that the number of labels in the model config does not match the number of labels in your dataset. You might need to manually update the model config to reflect the correct number of labels.

Installing additional dependencies: If certain models require additional dependencies like mmpretrain, you might need to create a custom environment that includes these dependencies. You can then use this custom environment to run your import pipeline.

Setting model-specific parameters: Some models might require specific parameters to be set in a certain way. For example, the cascade-mask-rcnn model seems to require the border_allowed parameter to be set to -1. You might need to manually adjust these parameters in the model config.
Clouddle 0 Reputation points

2024-02-28T10:32:40.69+00:00
Hi Yutong, thank you for your reply. I have already tried the methods you mentioned. But here's what I got.

Updating the number of labels in the model config. This error (number of labels are not updated in modelconfig) raised in azureml/acft/images/components/finetune/mmdetection/common/model.py, line 129, in set)num_labels. This is part of the finetune.py script which azure automl uses to do the finetuning. As a user, I can do nothing about it.

installing additional dependencies. This is not feasible, as importing mmdetection model is strictly using acft-mmdetection-image-gpu curated environment, because it is a mmdetection model import pipeline. I cannot use a custom environment to import the models.

Setting model-specific parameters. Please see https://github.com/azure/azureml-assets/wiki/components-mmdetection_image_objectdetection_instancesegmentation_finetune for all parameters that a user can specify.

Please kindly try this notebook and replace the model name 'maskrcnn_resnet50_fpn' with 'mask2former_r50_8xb2-lsj-50e_coco' in .set_training_parameters method, then you can reproduce the error. Seems like finetune.py cannot accomodate all models from mmdetection.

Cheers,

C
YutongTie-MSFT 46,976 Reputation points

2024-03-03T11:10:27.0333333+00:00

@Clouddle Thanks for your response, if this can not solve your problem, I would like to suggest you raising a support ticket, please let me know if you have no support plan, I am happy to enable you a free one for this case.

Regards,

Yutong
YutongTie-MSFT 46,976 Reputation points

2024-03-06T00:52:35.64+00:00

@Clouddle Please feel free to let me know if you need a support ticket, I am happy to enable you a free ticket for this case if needed. Thanks.