In #75528, we would like to tweak the structure of the tokens rustc uses. One of the components which heavily depends on the current structure is rustdoc (html and check_code_block_syntax module use lexer::StringReader).
I think, for this issue and in general, it makes sense to switch rustdoc over to rustc_lexer, which produces lower-level guaranteed lossless tokens, which look like a better fit for the task of syntax highlighting.
I plan to do this refactor myself, but I want to check with @rust-lang/rustdoc if this is a good idea.
EDIT: cc #63284
In #75528, we would like to tweak the structure of the tokens rustc uses. One of the components which heavily depends on the current structure is
rustdoc(htmlandcheck_code_block_syntaxmodule uselexer::StringReader).I think, for this issue and in general, it makes sense to switch
rustdocover torustc_lexer, which produces lower-level guaranteed lossless tokens, which look like a better fit for the task of syntax highlighting.I plan to do this refactor myself, but I want to check with @rust-lang/rustdoc if this is a good idea.
EDIT: cc #63284