Bugs in the Unified Tax Model for Document Intelligence

Question

I can't seem to open a support ticket for this, so I'd like to share my findings in how the Unified Tax Model has some bugs and hopefully get some guidance on what to do about it.

This is with API version: 2024-11-30 (4.0 General Availability)

Test 1 - Individual Documents

All individual documents are correctly identified except a Schedule 1, which always comes back incorrectly classified as a 1040 with no fields parsed. The only time a solo schedule 1 can be parsed correctly is when I use the Schedule 1 model.

Test 2 - Single Merged Tax Document

This is the most common use case you are going to find. I download my tax forms from a provider. It is the 1040 and multiple schedule documents based on my tax situation. In my test case situation, I have a 1040, Schedule 1, Schedule A, Schedule D, and Schedule SE.

When I use the unified model to process this document, the first one that comes back is Classified as Other. These pages include the 1040, Schedule 1, and Schedule A. None of these three are detected, split, or parsed correctly. The second document that comes back is the Schedule D, which is correctly identified. The Schedule SE is split into 2 documents, both classified as Schedule SE, with Page 1 and 2 being marked as separate documents.

When I use the 1040 model to process this merged tax document, it correctly identifies that I have 5 documents, but it classifies all of them as a 1040. Only the actual 1040 returns the parsed fields.

What am I supposed to do when the end user just uploads a single PDF with all their Tax documents in one file? Asking my clients to identify the document is out of the question. Multiple calls and fuzzy logic trying to fix a broken Microsoft service is not a cost-effective solution.

Answer 1

Hello Josh Christensen,

Thank you for posting your question in the Microsoft Q&A forum.

Thank you for sharing your detailed findings regarding the Unified Tax Model. It appears that there are inconsistencies in how the model handles individual and merged tax documents, particularly with Schedule 1 and multi-page tax forms. These issues can significantly impact the usability of the service, especially when end users upload a single PDF containing all their tax documents. 

Schedule 1 is incorrectly classified as a 1040 when processed individually using the Unified Tax Model. The Schedule 1 model works correctly but requires explicit usage. The Unified Tax Model fails to correctly identify, split, and parse merged tax documents (e.g., 1040, Schedule 1, Schedule A, Schedule D, Schedule SE). The 1040 model identifies the correct number of documents but misclassifies all of them as 1040.

You may take a structured approach as below:

• Ensure that the input PDFs are well-structured and meet the requirements for the Unified Tax Model. Verify that the documents are not scanned images or poorly OCR-processed files, as this can affect parsing accuracy. The link you may refer - https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview?view=doc-intel-4.0.0
• If the Unified Tax Model does not meet your needs, consider training a custom model using Azure Form Recognizer. Custom models allow you to define specific document types and improve accuracy for your use case. The link you may refer - https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/tax-document?view=doc-intel-4.0.0&source=recommendations
• As a temporary workaround, implement preprocessing logic to split merged tax documents into individual pages or sections before processing. Use the Schedule 1 model for Schedule 1 documents and the Unified Tax Model for other documents.

The issues you are encountering with the Unified Tax Model highlight the need for improvements in handling Schedule 1 and merged tax documents. While implementing preprocessing logic and using custom models can serve as temporary workarounds, escalating the issue to Microsoft Support and providing feedback are essential steps to ensure long-term resolution. 

 

If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue. Your contribution to the Microsoft Q&A community is highly appreciated.

Answer 2

I found a work around, but it requires making N+1 calls into the service where N is the number of tax forms in a single upload. This is NOT ideal because it makes using what is supposed to work out of the box into a much more costly service.

1. First, I send the document to the ClassifyDocumentAsync endpoint on the client using the model "prebuilt-tax.us.1040". This gives me back an accurate split of the documents and their page numbers and content, but classifies all of them as 1040.
2. I interrogate the lines for "Schedule 1", "Schedule A", etc using my own "fuzzy" classifier to fix the incorrect classification.
3. Using the pages and my fuzzy classification, I resubmit the file for each document found, providing specific pages and the specific model to the AnalyzeDocumentAsync endpoint on the client
4. I get back usable and correct results.

If I do step 1 with the unified model, it can't even parse the documents correctly. If MSFT fixes the unified model to be able to parse the documents accurately, I could just do this in one call.