Interface EncodedTokensProvider
Expand description
interface EncodedTokensProvider {
getInitialState(): IState;
tokenize?(line: string, state: IState): ILineTokens;
tokenizeEncoded(
line: string,
state: IState,
): IEncodedLineTokens;
}Methods§
§getInitialState(): IState
getInitialState(): IState
The initial state of a language. Will be the state passed in to tokenize the first line.
§tokenize?(line: string, state: IState): ILineTokens
tokenize?(line: string, state: IState): ILineTokens
Tokenize a line given the state at the beginning of the line.
§tokenizeEncoded(
line: string,
state: IState,
): IEncodedLineTokens
tokenizeEncoded(
line: string,
state: IState,
): IEncodedLineTokens
Tokenize a line given the state at the beginning of the line.
A "manual" provider of tokens, returning tokens in a binary form.