Clarify proc-macro token tree #2021
Conversation
This attempts to clarify some details about proc-macro tokens, in particular the differences in handling `_` from `macro_rules`. This includes several changes: - Removed the wordy description of a `macro_rules` TokenTree. This is already defined in the grammar in more detail. This does remove the sentence discussing metavariable substitutions. I would prefer to defer this to a more detailed description in the `macro_rules` chapter on how metavariable substitutions work. - Removed the English description of a proc-macro token tree, and instead write it as a grammar production. Closes rust-lang#2020 Closes rust-lang#1236
rustbot
added
the
S-waiting-on-review
| - Delimited groups (`(...)`, `{...}`, etc) | ||
| - All operators supported by the language, both single-character and | ||
| multi-character ones (`+`, `+=`). | ||
| - Note that this set doesn't include the single quote `'`. |
There was a problem hiding this comment.
I wrote the English text to highlight the differences between macro_rules and proc macro tokens, like multi-character operators (which are not a thing in proc macros), lifetimes, negative literals.
From the formal grammar the difference is very hard to see.
| | ProcMacroTokenPunct | ||
| | ProcMacroTokenLiteral | ||
| ProcMacroTokenIdent -> IDENTIFIER_OR_KEYWORD | RAW_IDENTIFIER | `_` |
There was a problem hiding this comment.
Doesn't reference consider _ a keyword?
I think it should, it's lexically an identifier, and it's reserved, so it's a keyword (== reserved identifier).
| When tokens are passed to a proc-macro: | ||
|
|
||
| - All multi-character punctuation is broken into single characters. | ||
| - `_` is treated as an identifier. |
There was a problem hiding this comment.
This is misleading, _ is a (reserved) identifier.
| - Negative literals are converted into two tokens (the `-` and the literal) | ||
| possibly wrapped into a delimited group ([`Group`]) with implicit delimiters | ||
| ([`Delimiter::None`]) when it's necessary for preserving parsing priorities. | ||
| - The `_` identifier is interpreted as the `_` punctuation. |
There was a problem hiding this comment.
I don't think the reference should ever call _ punctuation, it's a (reserved) identifier in all senses except one - ident matcher in macro_rules doesn't accept it for backward compatibility reasons (but it can likely be changed over an edition).
|
@rustbot author |
rustbot
added
S-waiting-on-author
|
Reminder, once the PR becomes ready for a review, use |
This attempts to clarify some details about proc-macro tokens, in particular the differences in handling
_frommacro_rules. This includes several changes:macro_rulesTokenTree. This is already defined in the grammar in more detail. This does remove the sentence discussing metavariable substitutions. I would prefer to defer this to a more detailed description in themacro_ruleschapter on how metavariable substitutions work.Closes #2020
Closes #1236