Clarification Needed: Retaining Labeled Data Across Versions in Custom Extraction Model

Question

Clarification Needed: Retaining Labeled Data Across Versions in Custom Extraction Model

Raj Tiwari 20

I’m working on a project using Azure Document Intelligence (Custom Extraction model), and I had a question regarding how labeled data is handled across model versions.

We initially trained a custom extraction model using a set of labeled documents and were able to test it successfully. Later, based on our testing results, we found the need to include a few additional document types and proceeded to label them using the same labeling project and layout.

However, after completing the new labeling and initiating a new training session, we noticed two things:

Only the newly labeled documents appeared in the label layout — the previously labeled documents from the original training set were no longer visible.

We were required to enter a new model name each time we trained, which resulted in the creation of a completely new model, rather than an updated version of the existing one.

It appears that the model is being retrained only on the new set of documents, and not on the full accumulated data — even though we are working within the same labeling project.

Since we expect to regularly update the model with new document formats over time, we were hoping there would be a way to keep the labeled data consolidated and continuously build upon the existing model, rather than creating disconnected versions each time.

Could you please clarify:

Is there a supported way to continuously update or extend an existing model with new labeled documents?
How can we ensure that the model gets trained on the entire labeled dataset, not just the newly added documents?
Is there a way to view or manage the full labeled dataset across training versions within the same project?
What’s the recommended best practice if we want to maintain and grow a single evolving model over time?

We’d appreciate your guidance, as this is a key part of our use case.

Thank you in advance for your support.

Prashanth Veeragoni 5,770 Reputation points Microsoft External Staff Moderator

2025-04-09T20:31:34.61+00:00

Hi Raj Tiwari,

Here I understood that your core concern is how to retain and reuse previously labeled documents across versions of the Custom Extraction model in Azure Document Intelligence

Here I am directly pointing out to your four queries:

1.Is there a supported way to continuously update or extend an existing model with new labeled documents?

No, not directly.

Azure Document Intelligence does not support incremental training or “transfer learning” for Custom Extraction models. Each training session starts fresh, so:

·       Every model is a new snapshot trained only on the documents provided during that training session.

·       There is no automatic reuse of previously labeled documents unless they are included again.

2.How can we ensure that the model gets trained on the entire labeled dataset, not just the newly added documents?

To train on the full set of documents (old + new), you need to:

·       Manually include both old and new labeled documents in your training dataset each time.

·       This means that the container or folder you specify for training should contain:

o   the original labeled documents

o   the new ones

o   the associated labels (label.json files)

Always maintain a centralized blob storage folder for labeled data — a kind of “master dataset” that grows over time.

3.Is there a way to view or manage the full labeled dataset across training versions within the same project?

Yes, but with limitations

Using Document Intelligence Labeling Tool, you can:

·       View all labeled documents that are still in your labeling project, but...

·       If the project gets modified or reset (e.g., you start a new model with just the new files), only those new documents show up unless you re-import or manually re-add the old ones.

Recommendation:

·       Use the "Export" function regularly to back up your labeled dataset (both documents and JSON labels).

·       Keep a consistent storage structure (e.g., /training-data/2024-01_batch/, /2024-04_new_docs/) and create a combined set for retraining.

4.What’s the recommended best practice if we want to maintain and grow a single evolving model over time?

Here you can follow below Strategy:

Hope I answered all your queries, do let me know if you have any further queries

Thank you!
Prashanth Veeragoni 5,770 Reputation points Microsoft External Staff Moderator

2025-04-10T14:23:34.65+00:00

Hi Raj Tiwari,

Following up to see if the above suggestion was helpful. And, if you have any further query do let me know.

Thank you!
Prashanth Veeragoni 5,770 Reputation points Microsoft External Staff Moderator

2025-04-11T13:20:31.03+00:00

Raj Tiwari,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thank you!