How to include custom and standard entities in Named Entity Recognition model

Beth O'Dwyer 0

I'm happy to train a NER model to detect a specific entity, however I would like to retain the pre-trained model's ability to detect standard entities like people, location, organisation etc. What's the best way to do this?

Saideep Anchuri 160 Reputation points Microsoft Vendor

2024-09-30T18:27:03.57+00:00

Hi Beth O'Dwyer,

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

To include both custom and standard entities in your Named Entity Recognition (NER) model while retaining the pre-trained model's capabilities, you can start by fine-tuning a pre-trained NER model that already recognizes standard entities such as people, locations, and organizations. By doing this, you can then train the model on your specific dataset that includes your custom entities.

Do let us know if that helps or have any further queries.

Thank You.
Saideep Anchuri 160 Reputation points Microsoft Vendor

2024-10-01T15:03:46.0766667+00:00

Hi Beth O'Dwyer

Following up to see if the given response was helpful.

Thank You.
Saideep Anchuri 160 Reputation points Microsoft Vendor

2024-10-03T14:59:27.6233333+00:00

Hi Beth O'Dwyer,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thank You.
Beth O'Dwyer 0 Reputation points

2024-10-04T08:12:16.46+00:00

Thank you for your response. I'm a bit unclear on this, so the fine-tuned NER model on azure should still be able to classify standard entities such as people, organizations etc even if they're not included in the fine-tuning dataset?
Saideep Anchuri 160 Reputation points Microsoft Vendor

2024-10-04T16:58:49.5066667+00:00

Hi Beth O'Dwyer,

Thank you for your follow-up query.

When you fine-tune a pre-trained Named Entity Recognition (NER) model in Azure, it will still recognize standard entities like people, organizations, and locations, even if those entities are not included in your fine-tuning dataset. The model retains its foundational knowledge from the original training, allowing it to continue identifying these standard entities.

Fine-tuning allows you to add custom entities while maintaining the ability to classify standard ones. Just make sure to evaluate the model's performance on a test dataset that includes both standard and custom entities to ensure it effectively recognizes all relevant types.

Thank you.

Share via

How to include custom and standard entities in Named Entity Recognition model

Your answer