Class Lexer

Extract tokens from the input string.

Hierarchy

Lexer

Implements

ILexer

Index

Constructors

constructor

new Lexer(defs: readonly Lexer.Definition[], options?: LexerBuildOptions): Lexer
Parameters
- defs: readonly Lexer.Definition[]
- Optional options: LexerBuildOptions
Returns Lexer
- Defined in src/lexer/lexer.ts:33

Properties

`Private` buffer

buffer: string

Only feed, reset can modify this var.

debug

debug: boolean

`Private` `Readonly` defs

defs: readonly Lexer.Definition[]

`Private` errors

errors: Token[]

Error token list.

`Private` lineChars

lineChars: number[]

How many chars in each line. Only update, reset can modify this var.

logger

logger: Logger

`Private` offset

offset: number

How many chars are digested. Only update, reset can modify this var.

`Private` rest

rest: string

The un-digested string. This var is used to cache the result of buffer.slice(offset) to prevent unnecessary string copy. Only feed, reset, update can modify this var.

`Private` trimmed

trimmed: boolean

Cache whether this lexer already trim start.

Accessors

digested

get digested(): number
How many chars are digested.

Returns number
Implementation of ILexer.digested
- Defined in src/lexer/lexer.ts:81

Methods

clone

clone(options?: {
debug?: boolean;
logger?: Logger;
}): Lexer
Clone a new lexer with the same state and definitions. If options.debug/logger is omitted, the new lexer will inherit from the original one.
Parameters
- Optional options: {
  debug?: boolean;
  logger?: Logger;
  }
  - Optional debug?: boolean
  - Optional logger?: Logger
Returns Lexer
Implementation of ILexer.clone
- Defined in src/lexer/lexer.ts:62

dryClone

dryClone(options?: {
debug?: boolean;
logger?: Logger;
}): Lexer
Clone a new lexer with the same definitions, without states. If options.debug/logger is omitted, the new lexer will inherit from the original one.
Parameters
- Optional options: {
  debug?: boolean;
  logger?: Logger;
  }
  - Optional debug?: boolean
  - Optional logger?: Logger
Returns Lexer
Implementation of ILexer.dryClone
- Defined in src/lexer/lexer.ts:55

feed

feed(input: string): Lexer
Append buffer with input.
Parameters
- input: string
Returns Lexer
Implementation of ILexer.feed
- Defined in src/lexer/lexer.ts:73

getErrors

getErrors(): readonly Token[]
Get error tokens.

Returns readonly Token[]
Implementation of ILexer.getErrors
- Defined in src/lexer/lexer.ts:363

getLineChars

getLineChars(): readonly number[]
Get how many chars in each line.

Returns readonly number[]
Implementation of ILexer.getLineChars
- Defined in src/lexer/lexer.ts:345

getPos

getPos(index: number): {
column: number;
line: number;
}
Get line number (starts from 1) and column number (starts from 1) from the index (starts from 0) of the input string.
Parameters
- index: number
Returns {
column: number;
line: number;
}
- column: number
- line: number
Implementation of ILexer.getPos
- Defined in src/lexer/lexer.ts:349

getRest

getRest(): string
Get the un-lexed string buffer.

Returns string
Implementation of ILexer.getRest
- Defined in src/lexer/lexer.ts:331

getTokenTypes

getTokenTypes(): Set<string>
Get all defined token types.

Returns Set<string>
Implementation of ILexer.getTokenTypes
- Defined in src/lexer/lexer.ts:339

hasErrors

hasErrors(): boolean
Returns boolean
Implementation of ILexer.hasErrors
- Defined in src/lexer/lexer.ts:367

hasRest

hasRest(): boolean
The rest of buffer not empty.

Returns boolean
Implementation of ILexer.hasRest
- Defined in src/lexer/lexer.ts:335

lex

lex(input?: string | Readonly<{
    expect?: Readonly<{
        text?: string;
        type?: string;
    }>;
    input?: string;
}>): null | Token
Try to retrieve a token. If nothing match, return null.

You can provide expect to limit the token type/content to be accepted.
Parameters
- input: string | Readonly<{
      expect?: Readonly<{
          text?: string;
          type?: string;
      }>;
      input?: string;
  }> = ""
Returns null | Token
Implementation of ILexer.lex
- Defined in src/lexer/lexer.ts:119

lexAll

lexAll(input?: string | {
input?: string;
stopOnError?: boolean;
}): Token[]
Try to retrieve a token list exhaustively.
Parameters
- input: string | {
  input?: string;
  stopOnError?: boolean;
  } = ""
Returns Token[]
Implementation of ILexer.lexAll
- Defined in src/lexer/lexer.ts:236

`Private` log

log(msg: string): void
Parameters
- msg: string
Returns void
- Defined in src/lexer/lexer.ts:40

`Private` res2token

res2token(res: Readonly<{
    accept: true;
    content: string;
    digested: number;
    error?: any;
    muted: boolean;
    rest: string;
}>, def: Lexer.Definition): Token
Parameters
- res: Readonly<{
      accept: true;
      content: string;
      digested: number;
      error?: any;
      muted: boolean;
      rest: string;
  }>
- def: Lexer.Definition
Returns Token
- Defined in src/lexer/lexer.ts:110

reset

reset(): Lexer
Returns Lexer
Implementation of ILexer.reset
- Defined in src/lexer/lexer.ts:44

take

take(n?: number): string
Take n chars from the rest of buffer and update state. This is useful when you have external logic to handle the token.
Parameters
- n: number = 1
Returns string
Implementation of ILexer.take
- Defined in src/lexer/lexer.ts:85

trimStart

trimStart(input?: string): Lexer
Remove ignored chars from the start of the rest of buffer.
Parameters
- input: string = ""
Returns Lexer
Implementation of ILexer.trimStart
- Defined in src/lexer/lexer.ts:260

`Private` update

update(digested: number, content: string, rest: string): Lexer
Parameters
- digested: number
- content: string
- rest: string
Returns Lexer
- Defined in src/lexer/lexer.ts:95

Class Lexer

Hierarchy

Implements

Index

Constructors

Properties

Accessors

Methods

Constructors

constructor

Parameters

defs: readonly Lexer.Definition[]

Optional options: LexerBuildOptions

Returns Lexer

Properties

Private buffer

debug

Private Readonly defs

Private errors

Private lineChars

logger

Private offset

Private rest

Private trimmed

Accessors

digested

Returns number

Methods

clone

Parameters

Optional options: { debug?: boolean; logger?: Logger; }

Optional debug?: boolean

Optional logger?: Logger

Returns Lexer

dryClone

Parameters

Optional options: { debug?: boolean; logger?: Logger; }

Optional debug?: boolean

Optional logger?: Logger

Returns Lexer

feed

Parameters

input: string

Returns Lexer

getErrors

Returns readonly Token[]

getLineChars

Returns readonly number[]

getPos

Parameters

index: number

Returns { column: number; line: number; }

column: number

line: number

getRest

Returns string

getTokenTypes

Returns Set<string>

hasErrors

Returns boolean

hasRest

Returns boolean

lex

Parameters

input: string | Readonly<{ expect?: Readonly<{ text?: string; type?: string; }>; input?: string; }> = ""

Returns null | Token

lexAll

Parameters

input: string | { input?: string; stopOnError?: boolean; } = ""

Returns Token[]

Private log

Parameters

msg: string

Returns void

Private res2token

Parameters

res: Readonly<{ accept: true; content: string; digested: number; error?: any; muted: boolean; rest: string; }>

def: Lexer.Definition

Returns Token

reset

`Optional` options: LexerBuildOptions

`Private` buffer

`Private` `Readonly` defs

`Private` errors

`Private` lineChars

`Private` offset

`Private` rest

`Private` trimmed

`Optional` options: {
debug?: boolean;
logger?: Logger;
}

`Optional` debug?: boolean

`Optional` logger?: Logger

`Optional` options: {
debug?: boolean;
logger?: Logger;
}

`Optional` debug?: boolean

`Optional` logger?: Logger

Returns {
column: number;
line: number;
}

input: string | Readonly<{
expect?: Readonly<{
text?: string;
type?: string;
}>;
input?: string;
}> = ""

input: string | {
input?: string;
stopOnError?: boolean;
} = ""

`Private` log

`Private` res2token

res: Readonly<{
accept: true;
content: string;
digested: number;
error?: any;
muted: boolean;
rest: string;
}>

`Private` update