Edit

Azure Content Understanding

ContentUnderstandingContextProvider analyzes file attachments with Azure Content Understanding and injects structured results into the agent context. It supports documents, images, audio, and video, including OCR, tables, structured fields, transcription, diarization, and segment summaries.

This integration uses the pre-processing pattern: it transforms incoming content before model invocation and can retain processed state for later turns.

For large documents, the provider can upload extracted markdown to a file-search vector store instead of placing the entire result in the model context.

Prerequisites

  • An Azure subscription.
  • Azure Content Understanding in a supported region.
  • The service's required model deployments.
  • Azure identity access to the resource.

Install the package

pip install agent-framework-azure-contentunderstanding --pre

Analyze a document

Attach ContentUnderstandingContextProvider to the agent and send a supported binary attachment. The provider removes the binary input after analysis and supplies the extracted content to the model.

async def main() -> None:
    credential = AzureCliCredential()

    # Set up Azure Content Understanding context provider
    cu = ContentUnderstandingContextProvider(
        endpoint=os.environ["AZURE_CONTENTUNDERSTANDING_ENDPOINT"],
        credential=credential,
        analyzer_id="prebuilt-documentSearch",  # RAG-optimized document analyzer
        max_wait=None,  # wait until CU analysis finishes (no background deferral)
    )

    # Set up the LLM client
    client = FoundryChatClient(
        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
        model=os.environ["FOUNDRY_MODEL"],
        credential=credential,
    )

    # Create agent with CU context provider.
    # The provider extracts document content via CU and injects it into the
    # LLM context so the agent can answer questions about the document.
    async with credential, cu:
        agent = Agent(
            client=client,
            name="DocumentQA",
            instructions=(
                "You are a helpful document analyst. Use the analyzed document "
                "content and extracted fields to answer questions precisely."
            ),
            context_providers=[cu],
        )

        # --- Turn 1: Upload PDF and ask a question ---
        # 4. Upload PDF and ask questions
        # The CU provider extracts markdown + fields from the PDF and injects
        # the full content into context so the agent can answer precisely.
        print("--- Upload PDF and ask questions ---")

        pdf_bytes = SAMPLE_PDF_PATH.read_bytes()

        response = await agent.run(
            Message(
                role="user",
                contents=[
                    Content.from_text(
                        "What is this document about? Who is the vendor, and what is the total amount due?"
                    ),
                    Content.from_data(
                        pdf_bytes,
                        "application/pdf",
                        # Always provide filename — used as the document key
                        additional_properties={"filename": SAMPLE_PDF_PATH.name},
                    ),
                ],
            )
        )
        usage = response.usage_details or {}
        print(f"Agent: {response}")
        print(f"  [Input tokens: {usage.get('input_token_count', 'N/A')}]\n")

Processing options

  • Leave analyzer_id unset to select a document, audio, or video search analyzer from the media type.
  • Set max_wait=None when the run must wait for analysis to complete.
  • Use FileSearchConfig for token-efficient retrieval over large extracted documents.
  • Reuse an AgentSession to preserve analyzed-document state across turns.

Next steps

Mistral

Go deeper:

  • Last updated on