Create a document processing custom model

After you review the requirements, you can get started creating your document processing model.

Sign in to AI Builder

  1. Sign in to Power Apps or Power Automate.

  2. On the left pane, select AI Builder > Explore.

  3. Select Extract custom information from documents.

  4. Select Get Started.

  5. A step-by-step wizard will walk you through the process by asking you to list all data you want to extract from your document. If you want to create your model by using your own documents, make sure you have at least five examples that use the same layout. Otherwise, you can use sample data to create the model.

  6. Select Train.

  7. Test the model by selecting Quick test.

Select the type of document

On the Choose document type step, select the type of document you want to build an AI model to automate data extraction. There are two options:

  • Structured and semi-structured documents. Structured and semi-structured documents are those where for a given layout, the fields, tables, checkboxes, and other items can be found in similar places. Examples of structured and semi-structured documents are invoices, purchase orders, delivery orders, tax documents, and more.

  • Unstructured and free-form documents. Unstructured documents are those where there's no set structure, usually free documents with a varying number of paragraphs. Examples of unstructured documents are contracts, statement of work, letters, and more.

    Screenshot of the tiles under Select the type of documents your model will process.

Define information to extract

On the Choose information to extract screen, define the fields, tables, and checkboxes you want to teach your model to extract. Select the +Add button to start defining these.

Screenshot of the step in the document processing wizard where to define the fields, tables, or checkboxes we want the AI model to extract.

  • For each Field, provide a name you'd like the field to have in the model.

  • For each Checkbox, provide a name you'd like the checkbox to have in the model. Define separate checkboxes for each item that can be checked in a document.

  • For each Table, provide the name you'd like the table to have. Also, define the different columns that the model should extract.

Group documents by collections

A collection is a group of documents that share the same layout. Create as many collections as document layouts that you want your model to process. For example, if you're building an AI model to process invoices from two different vendors, each having their own invoice template, create two collections.

Animation of creating collections.

For each collection that you create, you need to upload at least five sample documents per collection. Files with formats JPG, PNG, and PDF files are currently accepted.

Animation of uploading documents.

Note

You can create up to 200 collections per model.

Tag documents

By tagging the documents you've uploaded, you're teaching your AI Builder model to extract the fields and tables you've specified.

To start tagging, select one of the collections on the right panel.

Tag fields

To tag a field, draw a rectangle around the field you're interested in and select the field name that it corresponds to.

Animation of tagging fields in a document.

At any time, you can resize to adjust your selection.

If a field ends on one line and begins on another line, you can tag it using the ‘Continue tagging’ option.

Animation of tagging multiple words and line breaks.

You can also use the Control key in Windows or Command ⌘ in macOS to tag multiple words at once. You can also use the Shift key to select the first and last word to tag all words between the two.

When you hover over words in your documents, light blue boxes may appear. These indicate that you can draw a rectangle around those words to select a field.

Screenshot of selecting fields close up.

Tag tables

  1. Draw a rectangle around the table in the document you're interested in, and then select the table name that it corresponds to. The content of the panel on the right will change.

  2. Draw rows by left-clicking between row separators.

  3. Draw columns by pressing Ctrl + left-click.

  4. Once the rows and columns have been set, assign the headers to extract by selecting the header column and mapping it to the desired one.

  5. A preview of how the table will be extracted appears on the panel on the right.

  6. If the header of the table has been tagged, select Ignore first row so the header of the table isn't extracted as the table content.

The following animation illustrates the process:

Animation of tagging a table in a document.

An alternative way to define the rows and columns for a table is by selecting Delimit rows and columns at the top of the screen.

Screenshot of delimit rows and columns.

Use the advanced tagging mode

Advanced tagging mode allows you to tag tables at the cell level. Use this mode for complex tables like:

  • Tables that are skewed, where tagging with a grid isn't possible.
  • When you need to extract nested items, like an item within a cell.

Given the table from the following example, to extract the unit price, we'll define it as a separate column on the Choose information to extract step. We define Description, Unit price, Quantity, and Amount each as a column of the table and tag them accordingly using advanced tagging mode. See the animation below.

Animation of delimit rows and columns.

You can start tagging in the default mode to quickly capture all rows and columns. Then switch to advanced mode to adjust each cell and tag nested items.

Nested items in tables

You can tag items that are nested within a row by defining these as columns. Given the table from the example below, to extract the unit price, define it as a separate column on the Choose information to extract step earlier in this topic. Define Description, Unit price, Quantity, and Amount each as a column of the table and then tag them accordingly.

Screenshot of tagging nested items in tables.

Multipage tables

If a table spans across more than one page, make sure you tag it across all the pages. Once you've finished tagging the table on one page, select This table continues on next page and continue tagging the table on the following page. By doing this, you'll teach the model that the table can span across multiple pages.

Screenshot showing to select the option ‘This table continues on next page’ to continue tagging a table that spans across multiple pages.

Once you have tagged all the pages where the table is present. Select Done to mark the table as fully tagged.

You can tag complex tables that have merged cells, nested items, borders or no borders, or even unexpected elements like a coffee stain, as shown in the following example.

Screenshot showing to select the option ‘Done’ once the table has been tagged in all pages.

Tip

As a best practice, ensure that you upload documents for training with expected variations. For example, include samples where the entire table is on a single page and where tables span two or more pages.

Tag checkboxes

To tag a checkbox, draw a rectangle around the checkbox you're interested in extracting and select the checkbox name that it corresponds to.

Animation of tagging a checkbox.

If the quality of the document is low, AI Builder might not be able to detect the checkbox. If you're unable to tag a checkbox, do the following:

  1. On the panel on the right, select the three dots next to the checkbox you want to extract.

  2. Select Not available in document.

AI Builder supports detection and extraction of selection marks such as checkboxes and radio buttons, with different markers to indicate whether the selection is marked or not.

Field, checkbox, or table not in document

If a field, checkbox, or table isn't present in one of the documents you've uploaded for training, select Not available in document on the panel to the right, next to a field, checkbox, or table.

Animation of a field or table not in document.

Tag all documents across all collections

All the documents that you've uploaded are presented for you to tag. If you've created multiple collections, tag all documents across all the collections.

If fields can’t be tagged in your document, go to Fields couldn't be loaded for this document for possible resolutions.

Next step

Train and publish your document processing model

See also

Training: Process custom documents with AI Builder (module)