Module tokenizer 

Modules

operator

Macros

keywords 🔒

Structs

Token

TokenText

Interned text content for tokens, allowing processed string literals to be stored efficiently while keeping tokens Copy.

Tokenizer 🔒

Enums

Delimiter

Keyword

Skipped

Records tokens that were skipped before this token was issued.

TokenKind

Functions

dedent_multiline 🔒

Apply multiline string dedenting to raw source content.

is_op_char

process_escape_sequences 🔒

Process escape sequences in raw source text, producing the final string content.

tokenize

Type Aliases

CharIndices 🔒