TextChunker.SplitPlainTextParagraphs Method

  • Reference

Definition

Namespace:
Microsoft.SemanticKernel.Text
Assembly:
Microsoft.SemanticKernel.Core.dll
Package:
Microsoft.SemanticKernel.Core v1.0.1

Split plain text into paragraphs.

public static System.Collections.Generic.List<string> SplitPlainTextParagraphs (System.Collections.Generic.List<string> lines, int maxTokensPerParagraph, int overlapTokens = 0, string? chunkHeader = default, Microsoft.SemanticKernel.Text.TextChunker.TokenCounter? tokenCounter = default);
static member SplitPlainTextParagraphs : System.Collections.Generic.List<string> * int * int * string * Microsoft.SemanticKernel.Text.TextChunker.TokenCounter -> System.Collections.Generic.List<string>
Public Shared Function SplitPlainTextParagraphs (lines As List(Of String), maxTokensPerParagraph As Integer, Optional overlapTokens As Integer = 0, Optional chunkHeader As String = Nothing, Optional tokenCounter As TextChunker.TokenCounter = Nothing) As List(Of String)

Parameters

lines
List<String>

Lines of text.

maxTokensPerParagraph
Int32

Maximum number of tokens per paragraph.

overlapTokens
Int32

Number of tokens to overlap between paragraphs.

chunkHeader
String

Text to be prepended to each individual chunk.

tokenCounter
TextChunker.TokenCounter

Function to count tokens in a string. If not supplied, the default counter will be used.

Returns

List<String>

List of paragraphs.

Applies to