Azure Cognitive Search Indexer: How to recognize graphs and colors in PDF pie charts?

Question

I build a Azure skillset of an Indexer with the following skills:

1. "@odata.type": "#Microsoft.Skills.Util.DocumentExtractionSkill", to extract information from PDF;
2. "@odata.type": "#Microsoft.Skills.Vision.OcrSkill", to read .jpeg

   image stamped in a PDF;
   

3. "@odata.type": "#Microsoft.Skills.Text.MergeSkill", to merge content with the OcrSkill output;
4. "@odata.type": "#Microsoft.Skills.Text.SplitSkill", to splits the merge text into overlapping segments for vectorization
5. "@odata.type": "#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill", to vectorize the output of Text.MergeSkill;
6. "@odata.type": "#Microsoft.Skills.Text.KeyPhraseExtractionSkill". (for semantic search)

And I've built a Multi-Round Q&A on Your Data Azure promptflow for querying the documents in the Index, using gpt-4 and Ada.

Issue: As is show in the below picture,

the promptflow doesn't recognize graphs or colors in pie charts.

Questions:

• How can I solve this problem?
• Can be enough add #Microsoft.Skills.Vision.ImageAnalysisSkill?

More general questions:

• What is the best strategy (i.e., skillset) to interpret and understand various types of graphs, such as histograms, line charts, plots, and other similar visualizations?

Every comment is welcome and encourage.

Answer 1

Hello Mateo !

Thank you for posting your question on Microsoft Learn.

Yes, adding ImageAnalysisSkill can help. It allows you to analyze images for objects, text, and visual features (like colors and shapes). However, this skill alone may not fully extract numerical values or labels from a pie chart.

{
    "@odata.type": "#Microsoft.Skills.Vision.ImageAnalysisSkill",
    "visualFeatures": ["Categories", "Description", "Color", "Tags", "Objects"]
}

• Color: Helps to detect dominant colors.
• Objects: Can identify elements like "pie chart" or "bar graph".
• Tags: Provides descriptive labels like "finance", "diagram", etc.

To better extract structured data from graphs, use Custom Vision. You can train a model with example pie charts, histograms, and line graphs to recognize:

• Labels (for example "Banks", "Asset Managers")
• Proportions (for example 34%, 20%, etc.)
• Colors associated with categories

Azure Form Recognizer (Custom Model) can extract structured information from complex layouts:

• It can detect graph legends, values, and category labels.
• Supports tables, key-value pairs, and bounding boxes.

You can use Azure OpenAI GPT-4V (if available in your region) to reason about the extracted image content, drawing insights like:

• Matching colors to categories.
• Understanding relative proportions in pie charts.