How to avoid using setext in markdown output?

Sean Walsh 50

I'm running a document through Azure DI and then trying to split the markdown on headers, but I'm getting setext headers in the output, like so:

<!-- PageNumber=\"i | Page\" -->
EXECUTIVE SUMMARY
===
The purpose of this evaluation is to ...

The heading should be export as:

# EXECUTIVE SUMMARY

Is there a way I can get DI to output like this?

romungi-MSFT 43,696 Microsoft Employee

@Sean Walsh I think you are referring to guidance provided on this page. Does this help?

from langchain.text_splitter import MarkdownHeaderTextSplitter
...


headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]
text_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

docs_string = docs[0].page_content
splits = text_splitter.split_text(docs_string)

print("Length of splits: " + str(len(splits)))

Sean Walsh 50 Reputation points

2024-05-31T12:21:31.47+00:00

No, that's not it. I'm using Document Intelligence to produce the Markdown from the uploaded document. Sometimes it produces setext headings, which aren't recognised by the splitters, as that seems to only work with ATX headers. So I'd hoped I could get DI to only use ATX headers, and not setext ones.

Share via

How to avoid using setext in markdown output?