How can I debug the "Results do not correspond to current coco set" ?

Question

How can I debug the "Results do not correspond to current coco set" ?

Bolderink, Ingmar 0

Hello everybody,

I try to train a instance segmentation model and I run into the error "Results do not correspond to current coco set" (see below for more details).

I have formatted the input data for this model the same way as is described here: https://learn.microsoft.com/en-us/azure/machine-learning/reference-automl-images-schema. I even tried to generate the data the same way as an export of the labeling project in automl environment. Both give me the same results.

I have removed the validation metric type (coco_voc) and a run seems to succeed. I have previously trained a model with the coco_voc setting and I would like to be able to use the same settings with the new model.

I'm a little lost here: Has anybody encountered the same error and if so how did you solve it? Next tot that: what would be a good approach to debug this issue?

Thank you,

Ingmar

Error message:

Loading and preparing results...
Type: Unclassified
Class: AssertionError
Message: [Hidden as it may contain PII]
Traceback:   File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 315, in train_worker
    azureml_run=azureml_run)
  File "train.py", line 406, in train
    validation_evaluator.primary_metric, is_train=False)
  File "object_detection_utils.py", line 352, in compute_metrics
    coco_score = cocotools.score_from_index(val_coco_index, eval_bounding_boxes, task)
  File "cocotools.py", line 73, in score_from_index
    coco_detections = coco_index.loadRes(boxes)
  File "coco.py", line 327, in loadRes
    'Results do not correspond to current coco set'
ExceptionTarget: Unspecified
Type: Unclassified
Class: AssertionError
Message: [Hidden as it may contain PII]
Traceback:   File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 125, in run
    logger=logger, azureml_run=azureml_run)
  File "distributed_utils.py", line 418, in launch_single_or_distributed_training
    train_worker_fn(0, settings, *additional_train_worker_fn_args)
  File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 315, in train_worker
    azureml_run=azureml_run)
  File "train.py", line 406, in train
    validation_evaluator.primary_metric, is_train=False)
  File "object_detection_utils.py", line 352, in compute_metrics
    coco_score = cocotools.score_from_index(val_coco_index, eval_bounding_boxes, task)
  File "cocotools.py", line 73, in score_from_index
    coco_detections = coco_index.loadRes(boxes)
  File "coco.py", line 327, in loadRes
    'Results do not correspond to current coco set'
ExceptionTarget: Unspecified
Marking Run good_rabbit_mt6fz0nqsp_HD_2 as Failed.
Type: SystemError
Class: ClientException
Message: ClientException:
	Message: Encountered an internal AutoML Image error. Error Message/Code: {error_details}. Traceback:   File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 125, in run
    logger=logger, azureml_run=azureml_run)
  File "distributed_utils.py", line 418, in launch_single_or_distributed_training
    train_worker_fn(0, settings, *additional_train_worker_fn_args)
  File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 315, in train_worker
    azureml_run=azureml_run)
  File "train.py", line 406, in train
    validation_evaluator.primary_metric, is_train=False)
  File "object_detection_utils.py", line 352, in compute_metrics
    coco_score = cocotools.score_from_index(val_coco_index, eval_bounding_boxes, task)
  File "cocotools.py", line 73, in score_from_index
    coco_detections = coco_index.loadRes(boxes)
  File "coco.py", line 327, in loadRes
    'Results do not correspond to current coco set'
. Additional information: [Hidden as it may contain PII].
	InnerException: AssertionError
	ErrorResponse 
{
    "error": {
        "code": "SystemError",
        "message": "Encountered an internal AutoML Image error. Error Message/Code: {error_details}. Traceback:   File \"utils.py\", line 339, in wrapper\n    return func(*args, **kwargs)\n  File \"runner.py\", line 125, in run\n    logger=logger, azureml_run=azureml_run)\n  File \"distributed_utils.py\", line 418, in launch_single_or_distributed_training\n    train_worker_fn(0, settings, *additional_train_worker_fn_args)\n  File \"utils.py\", line 339, in wrapper\n    return func(*args, **kwargs)\n  File \"runner.py\", line 315, in train_worker\n    azureml_run=azureml_run)\n  File \"train.py\", line 406, in train\n    validation_evaluator.primary_metric, is_train=False)\n  File \"object_detection_utils.py\", line 352, in compute_metrics\n    coco_score = cocotools.score_from_index(val_coco_index, eval_bounding_boxes, task)\n  File \"cocotools.py\", line 73, in score_from_index\n    coco_detections = coco_index.loadRes(boxes)\n  File \"coco.py\", line 327, in loadRes\n    'Results do not correspond to current coco set'\n. Additional information: [Hidden as it may contain PII].",
        "details_uri": "https://aka.ms/automltroubleshoot",
        "inner_error": {
            "code": "ClientError",
            "inner_error": {
                "code": "AutoMLInternal",
                "inner_error": {
                    "code": "AutoMLVisionInternal"
                }
            }
        }
    }
}
Traceback:   File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 125, in run
    logger=logger, azureml_run=azureml_run)
  File "distributed_utils.py", line 418, in launch_single_or_distributed_training
    train_worker_fn(0, settings, *additional_train_worker_fn_args)
  File "utils.py", line 339, in wrapper
    return func(*args, **kwargs)
  File "runner.py", line 315, in train_worker
    azureml_run=azureml_run)
  File "train.py", line 406, in train
    validation_evaluator.primary_metric, is_train=False)
  File "object_detection_utils.py", line 352, in compute_metrics
    coco_score = cocotools.score_from_index(val_coco_index, eval_bounding_boxes, task)
  File "cocotools.py", line 73, in score_from_index
    coco_detections = coco_index.loadRes(boxes)
  File "coco.py", line 327, in loadRes
    'Results do not correspond to current coco set'

Search space:

image_instance_segmentation_job = automl.image_instance_segmentation(
        compute=compute_name,
        experiment_name=exp_name,
        training_data=my_training_data_input,
        validation_data=my_validation_data_input,
        target_column_name="label",
        primary_metric=InstanceSegmentationPrimaryMetrics.MEAN_AVERAGE_PRECISION,
        tags={"data_tag": "only enlabeler data"},
    )

    image_instance_segmentation_job.set_limits(
        timeout_minutes=60*100,
        max_trials=100,
        max_concurrent_trials=10,
    )

    image_instance_segmentation_job.extend_search_space(
        [
            SearchSpace(
                number_of_epochs= Choice([60]),
                model_name=Choice(["maskrcnn_resnet101_fpn"]),
                training_batch_size = Choice([3, 4, 5]),
                validation_batch_size = Choice([1]),
                learning_rate=Uniform(0.0001, 0.00035),
                warmup_cosine_lr_cycles=Uniform(.3, .6),
                warmup_cosine_lr_warmup_epochs=Choice([2, 3, 4]),
                optimizer=Choice(['adamw', 'adam']),
                min_size=Choice([750]),
                max_size=Choice([966]),
                validation_iou_threshold= Choice([0.8]),
                validation_metric_type= Choice(['coco_voc']),
                box_score_threshold= Choice([0.8])
            ),
        ]
    )

    image_instance_segmentation_job.set_sweep(
        sampling_algorithm="Random",
        early_termination=BanditPolicy(evaluation_interval=2, slack_factor=0.06, delay_evaluation=5),
    )

Ramr-msft 17,826 Reputation points

2023-02-21T05:20:08.78+00:00
The error message "Results do not correspond to current coco set" is usually encountered when the validation data provided does not match the expected format for the COCO evaluation metric. To debug this issue, you can check the following:

Ensure that the validation data is in the correct format as described in the Azure Machine Learning documentation.

Check if the validation data has the correct number of columns and if the columns are named correctly.

Ensure that the validation data contains the correct labels and annotations for the objects in the images.

If you are using a custom script to prepare the validation data, check if the script is generating the data correctly.

If you are using a labeling project in the Azure Machine Learning environment to generate the validation data, check if the labeling project is set up correctly and if the data is being exported correctly.

If you are still encountering the error after checking these points, you can try to remove the validation_metric_type hyperparameter and run the experiment again. This will allow you to see if the error is related to the validation metric type or if there is another issue with the data or the experiment setup.

Your answer

Ramr-msft 17,826 Reputation points

2023-02-21T05:20:08.78+00:00

The error message "Results do not correspond to current coco set" is usually encountered when the validation data provided does not match the expected format for the COCO evaluation metric. To debug this issue, you can check the following:

Ensure that the validation data is in the correct format as described in the Azure Machine Learning documentation.

Check if the validation data has the correct number of columns and if the columns are named correctly.

Ensure that the validation data contains the correct labels and annotations for the objects in the images.

If you are using a custom script to prepare the validation data, check if the script is generating the data correctly.

If you are using a labeling project in the Azure Machine Learning environment to generate the validation data, check if the labeling project is set up correctly and if the data is being exported correctly.

If you are still encountering the error after checking these points, you can try to remove the validation_metric_type hyperparameter and run the experiment again. This will allow you to see if the error is related to the validation metric type or if there is another issue with the data or the experiment setup.

Share via

How can I debug the "Results do not correspond to current coco set" ?

Your answer