Best Document Intelligence model for extracting fields when bill layouts and field positions change

Question

Best Document Intelligence model for extracting fields when bill layouts and field positions change

IT Cognity 0

Hello,

We are currently evaluating Azure AI Document Intelligence for extracting specific fields from electricity and natural gas bills.

Our main requirement is to build an extraction model that is robust to changes in the document layout, especially changes in the position of the fields we want to extract.

For example, a supplier may change the structure of its bill and move fields such as the supply number, meter number, billing period, or other required information to a different part of the document.

Ideally, we would like the model to identify and extract these fields primarily based on signals such as:

the field label/name,
the semantic meaning of the field,
the expected value structure or format,
and the general document context,

rather than relying heavily on the field's absolute position or the exact fields surrounding it.

PoC test

As a proof of concept, we trained a Custom Neural extraction model using sample documents.

We then modified one of the test documents by moving one of the required fields, together with its label, to a different position on the page.

The field itself, its label, and its expected value remained the same; only its position within the document changed.

When we analyzed the modified document, the model did not extract the field.

This raised some questions regarding the expected level of layout generalization of Custom Neural models.

Questions

How layout-independent is a Custom Neural extraction model expected to be? If a field that was included in the training dataset later appears in a significantly different position on the document, should a properly trained Custom Neural model still be able to identify and extract it?
What is the recommended training strategy for handling future field-position changes? Would it help to include multiple variations of the same document during training, where the required fields appear in different positions and with different surrounding layouts? For example, if the same field is shown in several different positions across the training documents, will the model learn to rely more on the field label, semantic context, and expected value structure instead of its location?
Would this type of training improve generalization to previously unseen layouts? Our objective is not only to support the layouts already included in the training set, but also to reduce the need for retraining whenever a supplier makes relatively small structural changes to its bill.
Is there any Microsoft guidance regarding the number and diversity of layout variations that should be included in the training dataset in order to achieve this?
Given this requirement, is Custom Neural the most appropriate Document Intelligence model? Or would another capability, such as Custom Generative extraction, be better suited when the extraction should depend more on the semantic meaning and expected structure of the fields and less on their visual location?

Our main objective is for changes in the location of a field within the bill to have minimal impact on extraction accuracy.

We understand that significant document changes may sometimes require retraining, but we would like to minimize that dependency and choose the model and training strategy that provide the best possible generalization across changing supplier layouts.

What would Microsoft recommend for this type of use case?

Thank you.

SRILAKSHMI C 19,730 Reputation points Microsoft External Staff Moderator

2026-09-04T14:13:33.1833333+00:00

Hello @IT Cognity

Thank you for the detailed description of your PoC and the example of the field-position change.

Based on your scenario, Custom Neural is the appropriate Document Intelligence custom model to evaluate first. Custom Neural models are designed to extract fields from structured and semi-structured documents and can generalize across different formats of the same document type. Unlike custom template models, they are not intended to require a separate model for every document variation.

However, a Custom Neural model should not be considered completely layout-independent. The model uses both language and layout information when learning how to extract labeled fields. Therefore, a significant change in the location or surrounding structure of a field can affect extraction accuracy, particularly if that variation was not represented in the training data.

Recommended training approach

For your electricity and natural-gas bill scenario, we recommend making the training dataset as representative as possible of the documents you expect in production.

Microsoft's guidance for Custom Neural models recommends:

Create a single model for variations of the same document type, where appropriate.

Include examples representing the different document variations you expect.

As a best practice, provide at least five labeled samples for each significant variation.

Use diverse and representative values for each field.

Use meaningful field names that correspond to the relevant value; for example, meter_number, supply_number, or billing_period.

Validate the trained model against documents containing variations that were not used during training.

Therefore, in your PoC, we recommend testing with multiple examples where the same field appears in different positions and with different surrounding layouts. This gives the model exposure to the expected variability and can improve its ability to generalize.

At the same time, there is no documented guarantee that a field can be moved arbitrarily anywhere on the page without affecting accuracy. If a supplier makes a substantial structural change, adding representative samples and creating a new model version may still be necessary. Microsoft also notes that there is no explicit "retrain" operation; training with an updated dataset generates a new model.

Custom Neural vs. Custom Generative extraction

For your specific requirement, I would recommend evaluating Custom Neural first if the bills are primarily structured or semi-structured and you have labeled examples of the fields you need.

Custom Neural is specifically intended for labeled custom extraction and supports structured and semi-structured documents such as invoices and similar business documents.

A generative extraction approach can be useful when the documents are highly variable or unstructured and the extraction requirement depends more heavily on semantic understanding or inferred fields. Microsoft positions Content Understanding custom analyzers for custom extraction without labels and for highly varied/unstructured documents.

For your bills, the recommended approach would therefore be:

Multiple representative bill layouts → label the required fields → train Custom Neural → test against unseen supplier/layout variations → measure field-level accuracy → add representative variations and train a new model version when necessary.

Regarding your specific PoC result

The fact that moving the field and its label caused the Custom Neural model not to extract it does not by itself indicate a service defect. It demonstrates that the particular layout variation was not sufficiently generalized by the trained model.

The best next step would be to expand the PoC dataset with several representative variations of the same bills, including different positions of supply_number, meter_number, billing_period, etc., and then compare the extraction accuracy on a separate test set.

Microsoft documentation:

Custom neural document model – Best practices

Custom document models

Choose the right Azure AI tool for document processing

I Hope this helps. Do let me know if you have any further queries.

Thank you!

1 answer

Your answer

Answer 1

The key issue is that Custom Neural models are not purely semantic models. They learn from a combination of text, language, and document layout signals. Therefore, moving a field and its label to a completely different location can impact extraction even when the label and value remain unchanged.

For your scenario, Custom Neural is generally the recommended starting point, as it is designed for structured, semi-structured, and unstructured documents and can generalise across document variations better than Template models. Microsoft also recommends starting with a Neural model where supported.

Regarding your specific questions:

Custom Neural is not fully layout-independent. Neural models use both language and layout features during training and inference. As a result, a field that appears in a significantly different location than the patterns observed during training may not always be extracted, even if the label and value remain unchanged. Your PoC result is therefore an expected behaviour rather than necessarily a model defect.
Including multiple real layout variations in training is the correct strategy. If suppliers use different bill formats, or if the same field appears in different locations across document versions, include those variations in the labelled training set. Neural models are specifically intended for documents that contain the same business information but differ in presentation and structure. [learn.microsoft.com]
Yes, greater layout diversity generally improves generalisation. While no model can guarantee extraction from completely unseen layouts, training with representative variations helps the model learn broader patterns instead of overfitting to a single document structure. [learn.microsoft.com], [learn.microsoft.com]
Microsoft's guidance is to include sufficient examples for each variation. A commonly recommended starting point is at least five labelled samples per layout variation, while larger and more diverse datasets typically produce better results.
Custom Generative extraction is worth benchmarking if layout volatility is expected to be high. Your requirement is fundamentally semantic:"find the supply number regardless of where it appears, provided the surrounding context indicates it is the supply number." In scenarios where suppliers frequently change layouts, Generative extraction may prove more resilient because it is oriented toward understanding document content rather than relying predominantly on learned document structure. However, the best approach is to benchmark it against Custom Neural using representative production documents rather than assuming it will automatically perform better.

Things to consider additionally:

One aspect that is often overlooked is that artificially moving fields within a document is not always representative of real-world layout evolution. When suppliers redesign bills, they usually change multiple elements simultaneously, including section headings, neighbouring content, table structures, typography, and page organisation. Training and testing with genuine supplier versions typically provides a more realistic measure of model robustness than synthetically relocating individual fields.

Overall, for your use case, I would recommend:

Custom Neural as the primary extraction model
Training with diverse, real supplier layouts and bill versions
Including multiple examples where key fields appear in different locations
Benchmarking Custom Generative extraction if supplier layouts change frequently and unpredictably

This approach provides the best balance between accuracy, maintainability, and resilience to future layout changes.

Help make this community better for everyone: if this answer resolved your issue, please accept it or leave an upvote. If not, share more details in a comment so we can continue the discussion and find the right solution.

Christos Lizos - Cognity 0 Reputation points

2026-09-04T14:25:36.9166667+00:00

Thank you for the clarification.

Could you also clarify the licensing and pricing requirements for Custom Generative extraction and Azure AI Content Understanding?

Do they require a different Azure subscription, license, SKU, or resource compared to the existing Azure AI Document Intelligence setup used for Custom Neural models, and is their pricing model different?

Thank you.