Custom Text Classification: How to train model and validate no matching labels

Question

Custom Text Classification: How to train model and validate no matching labels

Dylan Maxey 5

https://language.cognitive.azure.com/customText/projects/classification

I'm trying to train a model on text classification but can't seem to get the correct scores on the model preview, as the tests show under model performance that all the ones without a label are incorrect. I am importing the project via JSON. See documentation.

I tried multiple ways to get it to match. I tried using a the category "None", which is what's used as the default in the Language Studio, and that showed as no match. I tried "No prediction" which is what's used and the same happened. I tried

{"classes": []}

I also tried

{"classes": [{"category": null]}

as well as

{"classes": [{"category": ""]}

All of these attempts resulted as an import error.

If I go to the "Testing Deployments" tab and enter some text that doesn't get labeled, the json that gets returned is

{ "classes": [] }

Nothing seems to work, what am I missing here?

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-07-20T08:10:45.55+00:00

@Dylan Maxey I think the project settings has a value to set as a threshold to match to None of the labels below a threshold. It should be TextAnalysisAuthoringProjectSettings field of the settings.

The threshold of the class with the highest confidence, at which the prediction will automatically be changed to "None". The value of the threshold should be between 0 and 1 inclusive.

I think in your case you are trying to map a new label name to anything that does not match the rest of the labels, so you can use the response from JSON.

If you can set the value threshold from project settings and use the blank response in JSON as no match with some client-side processing it should also work. Hope this helps!!
Dylan Maxey 5 Reputation points

2023-07-20T15:23:32.7433333+00:00

@romungi-MSFT Here in the language studio, I can confirm the threshold is registered

However, under model performance, you can see it's still showing as a conflict.

This in turn directly affects our score for that specific category and as a result the model's total score.
Dylan Maxey 5 Reputation points

2023-07-24T13:38:42.7466667+00:00

@romungi-MSFT is there any insight you might be able to provide here?

Your answer

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-07-20T08:10:45.55+00:00

@Dylan Maxey I think the project settings has a value to set as a threshold to match to None of the labels below a threshold. It should be TextAnalysisAuthoringProjectSettings field of the settings.

The threshold of the class with the highest confidence, at which the prediction will automatically be changed to "None". The value of the threshold should be between 0 and 1 inclusive.

I think in your case you are trying to map a new label name to anything that does not match the rest of the labels, so you can use the response from JSON.

If you can set the value threshold from project settings and use the blank response in JSON as no match with some client-side processing it should also work. Hope this helps!!
Dylan Maxey 5 Reputation points

2023-07-20T15:23:32.7433333+00:00

@romungi-MSFT Here in the language studio, I can confirm the threshold is registered

However, under model performance, you can see it's still showing as a conflict.

This in turn directly affects our score for that specific category and as a result the model's total score.
Dylan Maxey 5 Reputation points

2023-07-24T13:38:42.7466667+00:00

@romungi-MSFT is there any insight you might be able to provide here?

Share via

Custom Text Classification: How to train model and validate no matching labels

Your answer