Create a document processing custom model
After you review the requirements, you can get started creating your document processing model.
Sign in to AI Builder
Sign in to Power Apps or Power Automate.
On the left pane, select AI Builder > Explore.
Select Extract custom information from documents.
Select Get Started.
A step-by-step wizard will walk you through the process by asking you to list all data you want to extract from your document. If you want to create your model by using your own documents, make sure you have at least five examples that use the same layout. Otherwise, you can use sample data to create the model.
Select Train.
Test the model by selecting Quick test.
Select the type of document
On the Choose document type step, select the type of document you want to build an AI model to automate data extraction. There are two options:
Structured and semi-structured documents. Structured and semi-structured documents are those where for a given layout, the fields, tables, checkboxes, and other items can be found in similar places. Examples of structured and semi-structured documents are invoices, purchase orders, delivery orders, tax documents, and more.
Unstructured and free-form documents. Unstructured documents are those where there's no set structure, usually free documents with a varying number of paragraphs. Examples of unstructured documents are contracts, statement of work, letters, and more.
Define information to extract
On the Choose information to extract screen, define the fields, tables, and checkboxes you want to teach your model to extract. Select the +Add button to start defining these.
For each Field, provide a name you'd like the field to have in the model.
For each Checkbox, provide a name you'd like the checkbox to have in the model. Define separate checkboxes for each item that can be checked in a document.
For each Table, provide the name you'd like the table to have. Also, define the different columns that the model should extract.
Group documents by collections
A collection is a group of documents that share the same layout. Create as many collections as document layouts that you want your model to process. For example, if you're building an AI model to process invoices from two different vendors, each having their own invoice template, create two collections.
For each collection that you create, you need to upload at least five sample documents per collection. Files with formats JPG, PNG, and PDF files are currently accepted.
Note
You can create up to 200 collections per model.
Tag documents
By tagging the documents you've uploaded, you're teaching your AI Builder model to extract the fields and tables you've specified.
To start tagging, select one of the collections on the right panel.
Tag fields
To tag a field, draw a rectangle around the field you're interested in and select the field name that it corresponds to.
At any time, you can resize to adjust your selection.
If a field ends on one line and begins on another line, you can tag it using the ‘Continue tagging’ option.
You can also use the Control key in Windows or Command ⌘ in macOS to tag multiple words at once. You can also use the Shift key to select the first and last word to tag all words between the two.
When you hover over words in your documents, light blue boxes may appear. These indicate that you can draw a rectangle around those words to select a field.
Tag tables
Draw a rectangle around the table in the document you're interested in, and then select the table name that it corresponds to. The content of the panel on the right will change.
Draw rows by left-clicking between row separators.
Draw columns by pressing Ctrl + left-click.
Once the rows and columns have been set, assign the headers to extract by selecting the header column and mapping it to the desired one.
A preview of how the table will be extracted appears on the panel on the right.
If the header of the table has been tagged, select Ignore first row so the header of the table isn't extracted as the table content.
The following animation illustrates the process:
An alternative way to define the rows and columns for a table is by selecting Delimit rows and columns at the top of the screen.
Use the advanced tagging mode
Advanced tagging mode allows you to tag tables at the cell level. Use this mode for complex tables like:
- Tables that are skewed, where tagging with a grid isn't possible.
- When you need to extract nested items, like an item within a cell.
Given the table from the following example, to extract the unit price, we'll define it as a separate column on the Choose information to extract step. We define Description, Unit price, Quantity, and Amount each as a column of the table and tag them accordingly using advanced tagging mode. See the animation below.
You can start tagging in the default mode to quickly capture all rows and columns. Then switch to advanced mode to adjust each cell and tag nested items.
Nested items in tables
You can tag items that are nested within a row by defining these as columns. Given the table from the example below, to extract the unit price, define it as a separate column on the Choose information to extract step earlier in this topic. Define Description, Unit price, Quantity, and Amount each as a column of the table and then tag them accordingly.
Multipage tables
If a table spans across more than one page, make sure you tag it across all the pages. Once you've finished tagging the table on one page, select This table continues on next page and continue tagging the table on the following page. By doing this, you'll teach the model that the table can span across multiple pages.
Once you have tagged all the pages where the table is present. Select Done to mark the table as fully tagged.
You can tag complex tables that have merged cells, nested items, borders or no borders, or even unexpected elements like a coffee stain, as shown in the following example.
Tip
As a best practice, ensure that you upload documents for training with expected variations. For example, include samples where the entire table is on a single page and where tables span two or more pages.
Tag checkboxes
To tag a checkbox, draw a rectangle around the checkbox you're interested in extracting and select the checkbox name that it corresponds to.
If the quality of the document is low, AI Builder might not be able to detect the checkbox. If you're unable to tag a checkbox, do the following:
On the panel on the right, select the three dots next to the checkbox you want to extract.
Select Not available in document.
AI Builder supports detection and extraction of selection marks such as checkboxes and radio buttons, with different markers to indicate whether the selection is marked or not.
Field, checkbox, or table not in document
If a field, checkbox, or table isn't present in one of the documents you've uploaded for training, select Not available in document on the panel to the right, next to a field, checkbox, or table.
Tag all documents across all collections
All the documents that you've uploaded are presented for you to tag. If you've created multiple collections, tag all documents across all the collections.
If fields can’t be tagged in your document, go to Fields couldn't be loaded for this document for possible resolutions.
Next step
Train and publish your document processing model
See also
Feedback
Submit and view feedback for