Why does the query fields function go away when selected output is markdown?

Question

Why does the query fields function go away when selected output is markdown?

Arman 50

This screenshot demonstrates that when markdown output is selected the Fields tab will disappear and you lose the query fields function:

Screenshot 2024-04-23 at 10.35.09 AM

When text is the selected output then the Fields tab appears again and the query field function is working.

Screenshot 2024-04-23 at 10.37.21 AM

What is this limitation about?

dupammi 8,615 Reputation points Microsoft External Staff

2024-04-23T08:24:03.62+00:00

Hi @Arman

Thank you for using the Microsoft Q&A forum.

The reason why the query fields function disappears when the markdown output is selected is that the markdown format simplifies the document's layout, making it easier for AI models to process the information. The markdown output is an intermediary format that bridges the gap between raw document data and structured data analysis. By converting document layouts into markdown, we can simplify the process of structuring document information before feeding it into AI models for extraction. However, this format can be difficult to use if you need to further harness this data by feeding it to Azure OpenAI, for large complex tables it may be quite verbose to be used inside a prompt.

On the other hand, if we use the plain text output, the table structure is getting lost. Therefore, markdown output is used as an intermediary format to bridge the gap between raw document data and structured data analysis.

For more details please refer - Analyze complex documents with Azure Document Intelligence Markdown Output and Azure OpenAI

I hope you understand.

Thank you.
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T01:20:24.2133333+00:00

Hi @Arman

We haven’t heard from you on the last response and was just checking back to see if you got a chance to see the information provided above.

Thank you.
Arman 50 Reputation points

2024-04-24T03:13:52.41+00:00

@dupammi Thank you for that reply. At least I know this isn't to do with a mistake on our side. The DI documentation certainly does not outline this limitation anyway but I think it would be good to update the docs to make this clear to users.

I think what you said can make sense in relation to Document Intelligence's (DI) implementation, however the expectation I describe in this forum question remains a valid use case. Ultimately DI is capable of resolving query fields and to also turn the document into markdown as separate functions. All that's missing is to return them both in the same response.

We have downstream processes that tremendously benefit from having the markdown version of the document and the query fields results. Your description attempts to justify this limitation on the basis of an implementation choice on DI's part but that reasoning does not explain away the limitation which this question is about. I also think that query fields and markdown responses are centerpieces of 2024-02-29-preview version but the two functions are not available together which again feels like a strange limitation.

Our workaround for this right now is to execute separate requests for the same document which incurs additional cost. Is there a possibility to have this limitation removed so a single requests can give us both markdown output and query fields results?
Arman 50 Reputation points

2024-04-24T03:28:41.39+00:00

(Sorry for double response. For some reason the first time I submitted it looked like nothing was submitted so I wrote my response one more time)

Thanks @dupammi . I now know this isn't a problem to do with how we are sending requests to Document Intelligence (DI). I suggest the documentation should be updated to outline this limitation.

From what you wrote, it appears that DI is limited this way due to its implementation choices?? (perhaps not and maybe you can shed more light on why) That is to say both query fields and markdown outputs exist as separate functions however, they are not allowed to be available together in a single response. This may justify why DI is that way right now but it's hard to see how it can explain away the desire to receive the markdown output and the query fields within the same response. Both markdown output and query fields are centerpieces of the 2024-02-29-Preview update but it's frustrating that both functions can not be available for a single request to analyze a document.

Our downstream processes tremendously benefit from having access to both of those capability. The workaround I can think of is to execute multiple requests against DI which is going to incur additional costs for us. Is there a chance that this limitation can be removed?
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T11:59:19.3066667+00:00

Hi @Arman

Thank you for your feedback and suggestion to add both query fields and markdown output in a single response in Document Intelligence Studio. Unfortunately, this is not currently available in Document Intelligence Studio. However, we appreciate your feedback and encourage you to submit a feature request for the same through the Azure Feedback Forum. Here's the link to the Azure Feedback Forum: Post idea · Community (azure.com)

In the meantime, the workaround you mentioned of executing multiple requests against DI is a viable option to think with additional costs incurred.

However, an optimization strategy that I can think of here, for getting both the markdown and query fields in a single request to DI is by passing the AnalyzeDocumentRequest object to the begin_analyze_document method with the output_content_format and features parameters set to ContentFormat.MARKDOWN and DocumentAnalysisFeature.QUERY_FIELDS, respectively, and specifying the query fields to be extracted using the query_fields parameter, the code should be able to perform both the markdown output and query fields extraction in a single request to DI.

This seems to be a viable solution to your problem of executing multiple requests against DI to get both markdown and query fields, which incurs additional costs, till the time your posted idea gets addressed by our team.

Below is the repro I tried at my end using python SDK for your reference. Kindly feel free to enhance as per your use case.

Repro: markdwn_Qfields_repro.txt

Output of repro:

Kindly refer these git repos for more implementation details: sample_analyze_documents_output_in_markdown.py

sample_analyze_addon_query_fields.py

Is this what you are looking for?
Arman 50 Reputation points

2024-04-24T15:47:25.55+00:00

@dupammi thanks. Perhaps I wasn't clear enough in my original post about what I am observing but this issue is not limited to Document Intelligence Studio. In fact I only described the problem in the context of Document Intelligence Studio because my original attempt at describing the issue with the REST API in another forum post seems to have got no tangible response.

I'm unfamiliar with Python and we are on Elixir but looking at the code you've provided makes me think it's no different to our code here which led us to discover the problem in the first place.

If I'm correct, your python code is simply setting the outputContentForm parameter to markdown. If so then I am 100% sure it does not work. Have you actually tested the python code you've provided to know if both markdown content and query fields are returned together in the same response?
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T16:06:23.93+00:00

Hi @Arman

Thank you for your response.

To extract query fields from the document, you need to set the features parameter to DocumentAnalysisFeature.QUERY_FIELDS, in addition to the output_content_format parameter. This is demonstrated in the code I provided earlier, which successfully extracts and prints the content in markdown format along with the detected query fields like address and invoice number.

I hope this gives clarity.

Thank you.

Arman 50

@dupammi since you insisted, I did set up a Python environment to test the sample you supplied. Maybe you confused the table from the PDF document with query fields in the script's output (perhapse because it's very similar to what the script wants to print from result.documents). After introducing the additional print statements, it's clear the code you've provided doesn't work.

i.e. the if condition in your code is never satisfied because result.documents is None:

print("......................")
print(result.documents)
print("......................")

if result.documents:
    for doc in result.documents:
        if doc.fields and doc.fields["Address"]:
            print(f"Address: {doc.fields['Address'].value_string}")
        if doc.fields and doc.fields["InvoiceNumber"]:
            print(f"Invoice number: {doc.fields['InvoiceNumber'].value_string}")

This is the output I get:

$ python markdwn_Qfields_repro.py                                                                                                               

Markdown format:
Contoso Address: 1 Redmond way Suite 6000 Redmond, WA 99243

Invoice For: Microsoft 1020 Enterprise Way Sunnayvale, CA 87659

| Invoice Number | Invoice Date | Invoice Due Date | Charges | VAT ID |
| - | - | - | - | - |
| 34278587 | 6/18/2017 | 6/24/2017 | $56,651.49 | PT |

......................
None
......................

dupammi 8,615 Reputation points Microsoft External Staff

2024-04-25T02:56:56.7733333+00:00

Hi @Arman

I understand that you experienced some inconvenience while trying out the code provided. It's great that you provided feedback, and I encourage you to submit a feature request through the Azure Feedback Forum. Here's the link to the forum: Post idea · Community.

I hope you understand.

Your answer

dupammi 8,615 Reputation points Microsoft External Staff

2024-04-23T08:24:03.62+00:00

Hi @Arman

Thank you for using the Microsoft Q&A forum.

The reason why the query fields function disappears when the markdown output is selected is that the markdown format simplifies the document's layout, making it easier for AI models to process the information. The markdown output is an intermediary format that bridges the gap between raw document data and structured data analysis. By converting document layouts into markdown, we can simplify the process of structuring document information before feeding it into AI models for extraction. However, this format can be difficult to use if you need to further harness this data by feeding it to Azure OpenAI, for large complex tables it may be quite verbose to be used inside a prompt.

On the other hand, if we use the plain text output, the table structure is getting lost. Therefore, markdown output is used as an intermediary format to bridge the gap between raw document data and structured data analysis.

For more details please refer - Analyze complex documents with Azure Document Intelligence Markdown Output and Azure OpenAI

I hope you understand.

Thank you.
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T01:20:24.2133333+00:00

Hi @Arman

We haven’t heard from you on the last response and was just checking back to see if you got a chance to see the information provided above.

Thank you.
Arman 50 Reputation points

2024-04-24T03:13:52.41+00:00

@dupammi Thank you for that reply. At least I know this isn't to do with a mistake on our side. The DI documentation certainly does not outline this limitation anyway but I think it would be good to update the docs to make this clear to users.

I think what you said can make sense in relation to Document Intelligence's (DI) implementation, however the expectation I describe in this forum question remains a valid use case. Ultimately DI is capable of resolving query fields and to also turn the document into markdown as separate functions. All that's missing is to return them both in the same response.

We have downstream processes that tremendously benefit from having the markdown version of the document and the query fields results. Your description attempts to justify this limitation on the basis of an implementation choice on DI's part but that reasoning does not explain away the limitation which this question is about. I also think that query fields and markdown responses are centerpieces of 2024-02-29-preview version but the two functions are not available together which again feels like a strange limitation.

Our workaround for this right now is to execute separate requests for the same document which incurs additional cost. Is there a possibility to have this limitation removed so a single requests can give us both markdown output and query fields results?
Arman 50 Reputation points

2024-04-24T03:28:41.39+00:00

(Sorry for double response. For some reason the first time I submitted it looked like nothing was submitted so I wrote my response one more time)

Thanks @dupammi . I now know this isn't a problem to do with how we are sending requests to Document Intelligence (DI). I suggest the documentation should be updated to outline this limitation.

From what you wrote, it appears that DI is limited this way due to its implementation choices?? (perhaps not and maybe you can shed more light on why) That is to say both query fields and markdown outputs exist as separate functions however, they are not allowed to be available together in a single response. This may justify why DI is that way right now but it's hard to see how it can explain away the desire to receive the markdown output and the query fields within the same response. Both markdown output and query fields are centerpieces of the 2024-02-29-Preview update but it's frustrating that both functions can not be available for a single request to analyze a document.

Our downstream processes tremendously benefit from having access to both of those capability. The workaround I can think of is to execute multiple requests against DI which is going to incur additional costs for us. Is there a chance that this limitation can be removed?
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T11:59:19.3066667+00:00

Hi @Arman

Thank you for your feedback and suggestion to add both query fields and markdown output in a single response in Document Intelligence Studio. Unfortunately, this is not currently available in Document Intelligence Studio. However, we appreciate your feedback and encourage you to submit a feature request for the same through the Azure Feedback Forum. Here's the link to the Azure Feedback Forum: Post idea · Community (azure.com)

In the meantime, the workaround you mentioned of executing multiple requests against DI is a viable option to think with additional costs incurred.

However, an optimization strategy that I can think of here, for getting both the markdown and query fields in a single request to DI is by passing the AnalyzeDocumentRequest object to the begin_analyze_document method with the output_content_format and features parameters set to ContentFormat.MARKDOWN and DocumentAnalysisFeature.QUERY_FIELDS, respectively, and specifying the query fields to be extracted using the query_fields parameter, the code should be able to perform both the markdown output and query fields extraction in a single request to DI.

This seems to be a viable solution to your problem of executing multiple requests against DI to get both markdown and query fields, which incurs additional costs, till the time your posted idea gets addressed by our team.

Below is the repro I tried at my end using python SDK for your reference. Kindly feel free to enhance as per your use case.

Repro: markdwn_Qfields_repro.txt

Output of repro:

Kindly refer these git repos for more implementation details: sample_analyze_documents_output_in_markdown.py

sample_analyze_addon_query_fields.py

Is this what you are looking for?
Arman 50 Reputation points

2024-04-24T15:47:25.55+00:00

@dupammi thanks. Perhaps I wasn't clear enough in my original post about what I am observing but this issue is not limited to Document Intelligence Studio. In fact I only described the problem in the context of Document Intelligence Studio because my original attempt at describing the issue with the REST API in another forum post seems to have got no tangible response.

I'm unfamiliar with Python and we are on Elixir but looking at the code you've provided makes me think it's no different to our code here which led us to discover the problem in the first place.

If I'm correct, your python code is simply setting the outputContentForm parameter to markdown. If so then I am 100% sure it does not work. Have you actually tested the python code you've provided to know if both markdown content and query fields are returned together in the same response?
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-24T16:06:23.93+00:00

Hi @Arman

Thank you for your response.

To extract query fields from the document, you need to set the features parameter to DocumentAnalysisFeature.QUERY_FIELDS, in addition to the output_content_format parameter. This is demonstrated in the code I provided earlier, which successfully extracts and prints the content in markdown format along with the detected query fields like address and invoice number.

I hope this gives clarity.

Thank you.
Arman 50 Reputation points

2024-04-25T01:50:18.1433333+00:00

@dupammi since you insisted, I did set up a Python environment to test the sample you supplied. Maybe you confused the table from the PDF document with query fields in the script's output (perhapse because it's very similar to what the script wants to print from result.documents). After introducing the additional print statements, it's clear the code you've provided doesn't work.

i.e. the if condition in your code is never satisfied because result.documents is None:

print("......................") print(result.documents) print("......................") if result.documents: for doc in result.documents: if doc.fields and doc.fields["Address"]: print(f"Address: {doc.fields['Address'].value_string}") if doc.fields and doc.fields["InvoiceNumber"]: print(f"Invoice number: {doc.fields['InvoiceNumber'].value_string}")

This is the output I get:

$ python markdwn_Qfields_repro.py Markdown format: Contoso Address: 1 Redmond way Suite 6000 Redmond, WA 99243 Invoice For: Microsoft 1020 Enterprise Way Sunnayvale, CA 87659 | Invoice Number | Invoice Date | Invoice Due Date | Charges | VAT ID | | - | - | - | - | - | | 34278587 | 6/18/2017 | 6/24/2017 | $56,651.49 | PT | ...................... None ......................
dupammi 8,615 Reputation points Microsoft External Staff

2024-04-25T02:56:56.7733333+00:00

Hi @Arman

I understand that you experienced some inconvenience while trying out the code provided. It's great that you provided feedback, and I encourage you to submit a feature request through the Azure Feedback Forum. Here's the link to the forum: Post idea · Community.

I hope you understand.

Share via

Why does the query fields function go away when selected output is markdown?

Your answer