Edit

Build and deploy custom document processing models on Azure

Azure AI Document Intelligence
Foundry Tools
Azure Logic Apps
Azure Machine Learning Studio
Microsoft Foundry
Azure Storage

This article describes Azure solutions for building, training, deploying, and using custom document processing models. These Azure services also provide user interface (UI) capabilities for labeling or tagging text during processing.

Architecture

Diagram that shows several alternatives for a custom document processing model build and deployment process.

Download a Visio file of this architecture.

Data flow

The following data flow corresponds to the previous diagram:

  1. Orchestrators like Azure Logic Apps, Azure Data Factory, or Azure Functions ingest messages and attachments from email servers and files from file transfer protocol servers or web applications.

    • Azure Functions and Logic Apps enable serverless workloads. The service that you choose depends on your preference for service capabilities like development, connectors, management, and operational context. For more information, see Compare Azure Functions and Logic Apps.

    • Consider using Azure Data Factory to move data in bulk.

  2. The orchestrators send ingested data to Azure Blob Storage or Azure Data Lake Storage. They organize the data within these stores based on characteristics like file extensions or customer details.

  3. You can use the following Azure services, either independently or in combination, for tagging documents and building custom models to address various use cases.

  4. To deploy the custom models and use them for inferencing:

    • Azure Document Intelligence in Foundry Tools has built-in model deployment. Inferencing with the custom models is done by using SDKs or document models REST API. The modelId, or model name, specified during model creation is included in the request URL for document analysis. Azure Document Intelligence doesn't require any further deployment steps.

    • Foundry provides an option to deploy custom language models. Get the REST endpoint prediction URL by selecting the model for deployment. You can run inference on models by using either the REST endpoint or the Azure SDK client libraries.

    • Machine Learning deploys custom models to online or batch Machine Learning managed endpoints. Models deployed through managed compute can be inferenced by using managed endpoints, which include online endpoints for real-time inferencing and batch endpoints for batch inferencing. These custom models can also be deployed on Azure Kubernetes Service (AKS) clusters.

    • Foundry provides multiple options to deploy fine-tuned Azure OpenAI models. You can deploy these models by using the Python SDK or REST API.

Components

  • Logic Apps is part of Azure Integration Services. Logic Apps creates automated workflows that integrate apps, data, services, and systems. In this architecture, Logic Apps orchestrates the ingestion of documents and data from various sources and triggers downstream processes for document processing. Use managed connectors for services like Azure Storage and Microsoft 365 to trigger workflows when a file arrives in the storage account or an email is received.

  • Azure Data Factory is a managed data integration service for orchestrating and automating data movement and transformation. In this architecture, Azure Data Factory adds transformation activities like invoking a REST endpoint or running a notebook on the ingested data to the pipeline.

  • Azure Functions is a serverless compute service that can host event-driven workloads that have short-lived processes. In this architecture, Functions enables workloads to process incoming documents and trigger model processing pipelines.

  • Blob Storage is an object storage solution for storing unstructured data. Blob Storage supports libraries for multiple languages, such as .NET, Node.js, and Python. Applications can access files on Blob Storage via HTTP or HTTPS. Blob Storage has hot, cool, and archive access tiers to support cost optimization for storing large amounts of data. In this architecture, this account is the solution for raw files that use a hot tier.

  • Data Lake Storage is a scalable, cloud-based repository for storing and organizing large volumes of unstructured data. In this architecture, Data Lake Storage organizes and maintains large volumes of ingested data to support analytics, labeling, and machine learning workflows.

  • Azure Document Intelligence is a component of Foundry Tools. In this architecture, it provides built-in document analysis capabilities for extracting printed and handwritten text, tables, and key-value pairs. Azure Document Intelligence has prebuilt models for extracting data from invoices, documents, receipts, ID cards, and business cards. Azure Document Intelligence also has a custom template form model and a custom neural document model that you can use to train and use custom models.

  • Document Intelligence Studio provides an interface to explore Azure Document Intelligence features and models. Use the interface to label data and build custom document processing models.

  • Azure Language provides natural language processing features like PII detection, language detection, prebuilt and customer NER, and text analytics for health.

  • Azure Machine Learning is a managed machine learning platform for model development and deployment at scale. In this architecture, it labels data, trains custom models (including with open-source frameworks), and deploys the models for inference tasks.

    • Machine Learning studio provides data labeling options for images and text. It supports model training workflows within this architecture.

    • Export labeled data as COCO or Machine Learning datasets. Use these datasets to train and deploy models in Machine Learning notebooks.

  • Azure OpenAI provides language and multimodal models through REST APIs. In this architecture, you fine-tune and deploy Azure OpenAI models for tasks such as text summarization and question answering.

Alternatives

This architecture includes multiple components that you can substitute with other Azure services or approaches, depending on your workload's functional and nonfunctional requirements. Consider the following alternatives and their trade-offs.

  • If the document is an image or a PDF, you can extract the data by using the Azure Document Intelligence Read API, Azure Content Understanding in Foundry Tools, or open-source libraries.

    • Azure Content Understanding is a service that uses generative AI to transform unstructured content across documents, audio, images, and video to structured outputs for multiple use cases like intelligent document processing, search and retrieval-augmented generation (RAG), robotic process automation (RPA), and analytics and reporting.

    • Use Azure Content Understanding Studio to create a custom analyzer by defining a field schema that extracts structured data from the document.

    • Create custom classifiers to classify documents into custom categories.

    • Route each category to a custom analyzer for field extraction. This approach combines classification and data extraction in a single pipeline.

  • Choose between Azure Content Understanding services and a custom solution based on your business requirements. For more information, see Choose the right tool for document processing.

  • Use the Foundry (classic) portal to fine-tune models for classification and custom text extraction from documents and to deploy these models.

Scenario details

Document processing covers a wide range of tasks. It can be difficult to meet all your document processing needs by using the prebuilt models available in Language and Azure Document Intelligence. You might need to build custom models to automate document processing for different applications and domains.

Major challenges in model customization include:

  • Labeling or tagging text data with relevant key-value pair entities to classify text for extraction.

  • Managing training infrastructure, such as compute and storage, and their integrations.

  • Deploying models at scale for applications to consume.

Potential use cases

The following use cases can take advantage of custom models for document processing:

  • Build custom NER and text classification models based on open-source frameworks.

  • Extract custom key values from documents for various industry verticals like insurance and healthcare.

  • Tag and extract specific domain-dependent entities beyond the prebuilt NER models for domains like security or finance.

  • Create custom tables from documents.

  • Extract signatures.

  • Label and classify emails or other documents based on content.

  • Summarize documents or create custom question-and-answer models based on your data.

Considerations

These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that you can use to improve the quality of a workload. For more information, see Well-Architected Framework.

For this example workload, implementing each pillar depends on optimally configuring and using each component Azure service.

Reliability

Reliability helps ensure that your application can meet the commitments that you make to your customers. For more information, see Design review checklist for Reliability.

Availability

  • To improve the availability of an Azure Document Intelligence-based solution, create two different resources in separate regions. Orchestrate either a failover to the other region or split workloads between the regions. Ensure that the custom models and custom classifiers are in sync in both regions. You can achieve this synchronization by copying these custom models from one Azure Document Intelligence resource to another in a supported region by using the Copy API.

  • When you build custom NER models by using Azure Language in Foundry Tools, replicate your project from one region to another supported region. Project replication copies only project settings and tagged data, so train and deploy the model in the secondary project. During a regional outage, route requests to that deployment. This approach helps ensure high availability (HA).

  • For more information about the service-level agreements for each architecture component, see Licensing documents.

  • For more information about configuration options to design highly available applications with Storage accounts, see Use geo-redundancy to design highly available applications.

Resiliency

  • Address failure modes of individual services like Functions and Storage to help ensure resiliency of the compute services and data stores in this scenario. For more information, see Reliability guides by service.

  • Machine Learning depends on constituent services like Blob Storage, compute services, and AKS. To provide reliability for Machine Learning, configure each of these services to be reliable. For more information on designing for recovery, see Failover for business continuity and disaster recovery (BCDR).

  • When you use Azure OpenAI models via Foundry, you must explicitly design resiliency by using deployments in multiple regions and failover strategies for HA and disaster recovery (DR). You can achieve this resiliency by deploying Azure OpenAI model endpoints across multiple Azure regions and by configuring explicit failover routing to maintain operations during a regional outage or capacity event.

Security

Security provides assurances against deliberate attacks and the misuse of your valuable data and systems. For more information, see Design review checklist for Security.

Implement data protection, identity and access management, and network security recommendations for Blob Storage, Foundry Tools for Azure Document Intelligence and Azure Language, Azure AI workloads for Machine Learning and Azure OpenAI.

Cost Optimization

Cost Optimization focuses on ways to reduce unnecessary expenses and improve operational efficiencies. For more information, see Design review checklist for Cost Optimization.

The total cost of implementing this solution depends on the custom models training and inferencing solution that you choose.

  • Azure Document Intelligence pricing includes the custom model training cost and the inferencing cost (for example, custom classification).

    If you build your own custom models for training, you pay the compute cost to train and deploy machine learning models. To help optimize costs, choose the right node type, cluster size, and number of nodes. Machine Learning provides options for training, such as setting the minimum number of compute cluster nodes to zero and defining the idle time before scaling down. For more information, see Manage and optimize Machine Learning costs.

  • Data orchestration duration and activities. For Azure Data Factory, the charges for copy activities on the Azure integration runtime are based on the number of data integration units used and the time taken to complete the activities. Added orchestration activity runs are also charged, based on their number.

    Logic Apps pricing plans depend on the resources that you create and use. The following articles can help you choose the right plan for specific use cases:

To estimate the cost of this solution, use this preconfigured estimate in the Azure pricing calculator. Adjust the values to match your expected document processing volumes, model training frequency, and inference workload.

Performance Efficiency

Performance Efficiency refers to your workload's ability to scale to meet user demands efficiently. For more information, see Design review checklist for Performance Efficiency.

Scalability

Contributors

Microsoft maintains this article. The following contributors wrote this article.

Principal authors:

  • Dixit Arora | Senior Engineer
  • Jyotsna Ravi | Sr. Account Executive

To see nonpublic LinkedIn profiles, sign in to LinkedIn.

Next steps