CodeTokenizer Class

Tokenize the code text into blocks.

Constructor

CodeTokenizer()

Methods

tokenize

Tokenize the code text into blocks.

tokenize

Tokenize the code text into blocks.

static tokenize(text: str) -> list[Block]

Parameters

Name	Description
text Required