Iteratively improving performance when using Content Understanding Classify and Generate fields

Question

Situation

I am exploring my options for automating a document processing workflow which is currently executed manually. It involves simple data extraction according to a schema (I've been blown away by Content Understanding's performance on this), but a trickier aspect too, which is to assess a number of criteria to check whether the document can be accepted for processing. The information relevant to these criteria can appear anywhere in the document - they can be satisfied by a statement, or a checkbox, or a check-mark on a row of a table.

The documents we may receive (it's insurance policy documents) can follow a large number of standardised formats, or not conform to any standardised format.

What I've tried

Content-Understanding only approach

I have been testing using Classify fields to encapsulate each criteria in the schema, like field name "CriteriaXSatisfied", with options "Yes", "No", "Unclear", accompanied with a Generate field which is asked to explain the reasoning behind that decision.

Again, the performance is good and on most of my test documents, the correct decision is made. I encountered one document where the decision was "No", and the reasoning was "Checkbox is not selected" - but it clearly is selected in the document. At this point I go into the labelling screen expecting to be able to provide this example, correct the labelling, and train a new analyzer which would have improved on the last. However it does not look like you can label the Classify or Generate types of field.

So it seems to me that the only way to try and improve performance is tweaking the descriptions in the analyzer schema. If that's the case, I anticipate playing a frustrating game where I may get one document to improve, only to introduce an issue in another.

Using an LLM with Vision

Another approach I am considering is using Content Understanding only for extracting data, and having a second step using an LLM to look at the document and assess my criteria. This approach would allow me to perform the iterative improvements via model fine-tuning I think, and I have seen some good results with this method.

However, there are the drawbacks of additional cost/run-time (we would ideally be processing these documents on upload to a portal, so reducing run-time would be good).

Document Intelligence

Early-on I was experimenting with Azure Document Intelligence, however this didn't seem well-suited to handling the large number of document templates we need to be able to handle (without a lot of effort labelling data for each possible one), and wouldn't offer any possibility to handle unexpected templates. I've discounted it on these grounds, but thought it worth mentioning in case someone might suggest a way around these limitations?

Question

Is Content Understanding the right tool for the job, given the requirements I've outlined? Any suggestions on this would be welcome.

I appreciate that Content Understanding is only in Preview currently, so I appreciate the answer may be that what I need (labelling Classify & Generate field methods) may be on the roadmap and we'll just have to wait

Answer 1

Hi Ewan Davies

Based on your current workflow and requirements, Microsoft Content Understanding is a strong candidate for structured data extraction and basic classification tasks. Your approach of using Classify fields to assess criteria and Generate fields to explain decisions is well-aligned with the tool’s capabilities. However, a key limitation is that these field types cannot be labeled or corrected, which restricts your ability to iteratively improve performance through retraining. Microsoft currently recommends refining field descriptions to guide the model more effectively, but this method can be brittle and may not generalize well across diverse document formats (Microsoft Best Practices).

Given the variability in your insurance policy documents and the nuanced nature of your criteria—such as interpreting checkboxes, table rows, or embedded statements—a hybrid architecture is recommended. In this setup, Content Understanding would handle structured extraction, while a vision-language model like GPT-4V would evaluate complex, visual, or context-dependent criteria. This approach allows for fine-tuning or retrieval-augmented generation (RAG), enabling iterative improvements that Content Understanding alone cannot support (Classifier Concepts).

To enhance this further, you could integrate lightweight vision models (e.g., YOLO) for tasks like checkbox detection and apply confidence thresholds to determine when to escalate to the LLM. While Azure Document Intelligence is another option, it typically requires extensive template-specific training and may not generalize well to unexpected formats. Overall, Content Understanding provides a solid foundation, but for more complex and variable document processing, combining it with a vision-capable LLM offers greater flexibility and accuracy. Microsoft may eventually support labeling for Classify and Generate fields, but this feature is not currently available (Microsoft Q&A).

Thanks