Hello Everyone, I am building Bot to do Question Answering over PDF with embedded Images, (not OCR). The document is like a user guide/manual. The answer will be a text response showing the respective image(s). What is the right way to preprocess the document?. There are Images without descriptions , is there way to map these images to respective sections in the document when extracting? Thank You

QA over PDF with Images

Jason Isaac 0

Hello Everyone,

I am building Bot to do Question Answering over PDF with embedded Images, (not OCR). The document is like a user guide/manual.

The answer will be a text response showing the respective image(s).

What is the right way to preprocess the document?.
There are Images without descriptions , is there way to map these images to respective sections in the document when extracting?

Thank You

Dillon Silzer 56,681 Reputation points

2023-10-19T05:12:22.43+00:00

Just to gather some more information, these PDFs are just filled with images or is there text in them as well? How are the PDFs formatted?

Also, have you taken a look at Azure Computer Vision API? I'd recommend reading up on this.
Jason Isaac 0 Reputation points

2023-10-19T05:45:04.2033333+00:00

Hi @Dillon Silzer ,

Thanks for the quick reply.

just filled with images or is there text in them as well?

The actual document is similar to below image. So, yeah it is text with images. No need to extract what is inside the image.

e.g, when there is query about features and datasets, bot should show this Figure 12 along with text answer.

How to create a KB and manage my text and these images?.

Additional challenge is the document I am working with doesn't have the description like Figure 12.

How are the PDFs formatted?

Hmm.. How to find that?.

Sure, I will see the Azure Computer Vision API, but I don't want to do any image processing, will it help solve my current problem?
YutongTie-MSFT 48,586 Reputation points

2023-10-31T06:26:38.97+00:00

@Jason Isaac Thanks for reaching out to us, if your image contains text and you do want to extract text, you may need to use Azure Computer Vision API like OCR to do so.

First, you can use OCR to extract the text from the document, including any text that is embedded in the images. This will allow you to perform text-based question answering on the document.

Next, you can use image analysis techniques to extract information about the images in the document, such as their location, size, and content. This will allow you to map the images to their respective sections in the document and provide a text response showing the respective image(s) when answering questions.

If there are images without descriptions, you can use image analysis techniques to identify the content of the images and generate descriptions automatically. For example, you can use computer vision algorithms to identify objects, scenes, and other visual features in the images and generate descriptive text based on these features.

Please take a look at above to see if that is something you are looking for, I hope this helps.

Share via

QA over PDF with Images