Semantic tokens

Semantic tokens provide compiler-level symbol classification to code editors, enabling intelligent, context-aware syntax highlighting that transcends basic regex matching.

Semantic tokens deliver a critical layer of accuracy to code editors, moving beyond the limitations of standard TextMate grammars which rely on lexical, regex-based rules. This technology leverages the Language Server Protocol (LSP) to transmit rich, project-contextual data from the language service to the editor. For example, a standard grammar might color all `variable` names the same; a semantic token, however, can correctly distinguish a local `parameter` from a global `class` or `function` declaration. This compiler-informed understanding allows themes to apply precise styling, significantly improving code readability and developer experience.

https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide