Share via

Tokenizer<TSymbol, TSymbolType> Class

  • Article

This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.

Inheritance Hierarchy

System.Object
  System.Web.Razor.StateMachine<TSymbol>
    System.Web.Razor.Tokenizer.Tokenizer<TSymbol, TSymbolType>
      System.Web.Razor.Tokenizer.CSharpTokenizer
      System.Web.Razor.Tokenizer.HtmlTokenizer
      System.Web.Razor.Tokenizer.VBTokenizer

Namespace:  System.Web.Razor.Tokenizer
Assembly:  System.Web.Razor (in System.Web.Razor.dll)

Syntax

'Declaration
Public MustInherit Class Tokenizer(Of TSymbol As SymbolBase(Of TSymbolType), TSymbolType) _
    Inherits StateMachine(Of TSymbol) _
    Implements ITokenizer
'Usage
Dim instance As Tokenizer(Of TSymbol, TSymbolType)

public abstract class Tokenizer<TSymbol, TSymbolType> : StateMachine<TSymbol>, 
    ITokenizer 
where TSymbol : SymbolBase<TSymbolType>

generic<typename TSymbol, typename TSymbolType>
where TSymbol : SymbolBase<TSymbolType>
public ref class Tokenizer abstract : public StateMachine<TSymbol>, 
    ITokenizer

[<AbstractClassAttribute>]
type Tokenizer<'TSymbol, 'TSymbolType when 'TSymbol : SymbolBase<'TSymbolType>> =  
    class 
        inherit StateMachine<'TSymbol>
        interface ITokenizer 
    end

JScript does not support generic types and methods.

Type Parameters

  • TSymbol
    The type for the language symbol.
  • TSymbolType
    The enumeration type for the language symbol.

The Tokenizer<TSymbol, TSymbolType> type exposes the following members.

Constructors

  Name Description
Protected method Tokenizer<TSymbol, TSymbolType> This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Initializes a new instance of the Tokenizer<TSymbol, TSymbolType> class.

Top

Properties

  Name Description
Protected property Buffer This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets or sets the buffer for the tokenizer.
Protected property CurrentCharacter This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the current character in the tokenizer.
Protected property CurrentErrors This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets a list of the current razor errors.
Protected property CurrentLocation This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the current source location.
Protected property CurrentStart This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the current start of the source location.
Protected property CurrentState This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets or sets the current state of the machine. (Inherited from StateMachine<TReturn>.)
Protected property EndOfFile This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets a value whether the tokenizer current location is at the end of the file.
Protected property HaveContent This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets a value whether the tokenizer have content.
Public property RazorCommentStarType This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the star type for the razor comment.
Public property RazorCommentTransitionType This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the transition type for the razor comment.
Public property RazorCommentType This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the type of razor comment.
Public property Source This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the source of the text document.
Protected property StartState This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Gets the starting state of the machine. (Inherited from StateMachine<TReturn>.)

Top

Methods

  Name Description
Protected method AfterRazorCommentTransition This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Returns a result after the razor comment transition.
Protected method At This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Determines whether the lookahead buffer contains the expected string.
Protected method CharOrWhiteSpace Returns a function delegate, that accepts a character parameter and returns a value that indicates whether the character parameter is equal to specified character or white space.
Protected method CreateSymbol This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Creates a language symbol type for the tokenizer with the specified content.
Protected method EndSymbol(TSymbolType) This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Returns the language end symbol type used by the tokenizer.
Protected method EndSymbol(SourceLocation, TSymbolType) This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Returns the language end symbol type used by the tokenizer.
Public method Equals (Inherited from Object.)
Protected method Finalize (Inherited from Object.)
Public method GetHashCode (Inherited from Object.)
Public method GetType (Inherited from Object.)
Protected method MemberwiseClone (Inherited from Object.)
Protected method MoveNext This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Reads to the next character from the code reader.
Public method NextSymbol This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Shows the next symbol to be used.
Protected method Peek This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Reads the next symbol in the code.
Protected method RazorCommentBody This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Parses the Razor comment body.
Public method Reset This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Sets the tokenizer status to its initial state.
Protected method ResumeSymbol This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Resumes using the previous language symbol type.
Protected method Single This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Uses a single type of symbol.
Protected method StartSymbol This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Returns the start symbol used in this class.
Protected method Stay() This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Stays into the machine during the transition. (Inherited from StateMachine<TReturn>.)
Protected method Stay(TReturn) This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Stays into the machine during the transition with the specified output. (Inherited from StateMachine<TReturn>.)
Protected method Stop This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Disables the machine upon transition. (Inherited from StateMachine<TReturn>.)
Protected method TakeAll This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Takes the string if found in the lookahead buffer into the tokenizer buffer.
Protected method TakeCurrent This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Accepts the current character into the buffer.
Protected method TakeString This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Accepts the given input string into the buffer.
Protected method TakeUntil Parses the source document until the condition specified by predicate is met or end file is reached.
Public method ToString (Inherited from Object.)
Protected method Transition(StateMachine<TReturn>.State) This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Indicates the new transition of the state. (Inherited from StateMachine<TReturn>.)
Protected method Transition(TReturn, StateMachine<TReturn>.State) This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Indicates the new transition of the state with the specified output. (Inherited from StateMachine<TReturn>.)
Protected method Turn This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Describes the turning process of the state. (Inherited from StateMachine<TReturn>.)

Top

Explicit Interface Implementations

  Name Description
Explicit interface implemetationPrivate method ITokenizer.NextSymbol This type/member supports the .NET Framework infrastructure and is not intended to be used directly from your code.Returns the next language symbol type.

Top

Thread Safety

Any public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe.

See Also

Reference

System.Web.Razor.Tokenizer Namespace