SwiftSyntax support for module selectors #3091

beccadax · 2025-06-02T19:44:22Z

This PR adds module selectors to SwiftSyntax and SwiftParser (draft of matching compiler feature in swiftlang/swift#34556). This feature was pitched ages ago; a proper proposal is ~~on my todo list~~ now available.

Note: This PR is still a work in progress—in particular, I need to go back and improve the tests, both by expanding their coverage and by adding assertions about the resulting syntax trees—but I'd appreciate feedback on the design while I'm working on that.

Reviewers: If you review this commit-by-commit, I've separated out the big dumb mechanical changes from the ones that actually change parsing logic.

nkcsgexi · 2025-06-02T19:47:56Z

🥳

beccadax · 2025-06-02T20:04:18Z

Sources/SwiftParser/Names.swift

+  ///
+  /// - Precondition: `node` must, at minimum, have a descendant with an unexpected nodes child; it therefore cannot be
+  ///   a token or an empty collection.
+  func attach<Node: RawSyntaxNodeProtocol>(_ moduleSelector: RawModuleSelectorSyntax?, to node: Node) -> Node {


Retroactively rewriting a node to insert new information is a pretty unconventional way for SwiftParser to do things, so I thought I should justify this design.

Because a module selector is a prefix on a different syntax and requires two tokens, it's difficult to peek past one and make decisions based on the tokens that follow it. Instead, I found it easier to parse them in relatively high-level productions; that gets the tokens out of the way, but it means they're often parsed a significant distance from the node that will become their parent. I tried two other approaches to coping with this problem:

Adding new ModuleSelectorExprSyntax and ModuleSelectorTypeSyntax nodes. The issue was that I want to make sure we could parse invalid module selectors—even in declaration syntax, patterns, etc.—into unexpected nodes, and that didn't offer a good way to do so. (I also found that the module selectors on member lookups wouldn't be able to be handled in this way; I didn't really like that.)

Passing the ModuleSelectorSyntax node down as a parameter and threading it through to wherever it was needed. This required me to add a lot of parameters and manually insert unexpected module selectors into a lot of nodes, but that's not actually what scuttled the idea—it's that it dramatically increased stack usage. At one point I had to reduce the maximum recursion in development builds to 10, which isn't even enough to handle the swift-syntax repo itself.

Retroactively attaching module selectors in this fashion allows more of the parser to ignore the fact that an invalid module selector might have been parsed earlier on, while also avoiding the stack usage problems I mentioned.

Could you explain why we want

I want to make sure we could parse invalid module selectors—even in declaration syntax, patterns, etc.

What makes module selectors different from other invalid things? E.g.:

let Foo.Bar = 12

A couple reasons:

The difference between a name and a name reference is a bit subtle and hasn't previously been grammatically important; I can imagine that developers might be confused about where they are allowed to use module selectors.

If a developer writes Foo::Bar in a place where module selectors are not valid, we can be almost certain that they want to keep Bar (because we know that Foo was supposed to be a module name), but the naïve recovery behavior will treat Foo as the name and ::Bar as unexpected syntax. Tailoring the recovery gives us a tree that better reflects the user's likely intent. (Whereas with let Foo.Bar the developer might have wanted let Bar, or let Foo, or let Foo = .Bar, or any number of other possibilities; it's harder to be certain of their intent.)

I've been thinking about this. but I'm still not convinced. Let's say for let Foo::Bar = <expr>, I don't think we can say the user tried to specify the module name. The user might mistyped let Foo: Bar = <expr>.

Whether developers might get confused between name declarations and name references is subjective, so I won’t argue that. But I don’t think it’s worth adding this extra complexity to the implementation. It looks overly complex. Also, rewriting parsed nodes leaves abandoned nodes in the arena, which is not great for memory usage. IMO, the module selector parsing should be mostly contained in parseDeclReferenceExpr(), (can)parseTypeIdentifier(), and (can)parseSimpleType() (for member types). Of course we should do our best to emit better diagnostics. But I don't feel this is the way. Also we can improve diagnostics later. Can we make it simple for the initial implementation?

Also, rewriting parsed nodes leaves abandoned nodes in the arena, which is not great for memory usage

Yeah, our policy so far has been that we're okay with having abandoned nodes in the error case but not the success. I agree with Rintaro here - the added complexity doesn't seem worth the difference in the diagnostic (and tree) that we'd end up with.

This capability is currently unused, but it’ll matter soon.

And make sure it doesn’t break Objective-C selector lexing.

Treating introducers as keywords is now always conditional on whether `shouldParsePatternBinding(introducer:)` returns `true`. That method has also been modified to correctly handle an edge case with wildcard patterns.

Initializers for nodes with experimental node children need to be marked `@_spi`. This PR: • Adds that attribute. • Generates an alternative which *doesn’t* use SPI as part of the compatibility layer. • As a side effect, adds a `Child.Refactoring.introduced` case that can be used to generate compatibility `unexpected` properties. No functional change in this commit, but it will affect the code generation in the next one.

beccadax · 2025-07-10T03:10:31Z

@swift-ci please test

beccadax · 2025-07-11T00:11:22Z

@swift-ci please test

beccadax · 2025-07-11T20:50:43Z

@swift-ci please test

Changes the syntax tree to represent module selectors: • A `ModuleSelectorSyntax` node represents a module selector abstractly. • The following nodes now have an optional `moduleSelector` child: • `DeclReferenceExprSyntax` • `IdentifierTypeSyntax` • `MacroExpansionExprSyntax` • `MemberTypeSyntax` • BasicFormat knows the preferred format for module selectors. Other components, particularly the parser, were also updated to continue building, though without any changes in behavior. Parser implementation will come in a future commit.

Changes it to share code with `parseTypeIdentifier()` and clean up the member type parsing a little. Also tweaks call sites of `parseTypeIdentifier()`.

This commit ports over tests from the compiler’s (future) `test/NameLookup/module_selector.swift` file and makes sure the correct uses parse as expected. It also tests that ill-formed module selectors (ones with a missing or non-identifier module name) are diagnosed correctly. This commit doesn’t fully handle recovery from module selectors inserted at invalid locations; the test cases that require recovery are XFAILed.

Specifically, from module selectors at incorrect locations. This is done through a couple of mechanisms: • The various `expect(…)` methods consume a module selector as unexpected syntax. • Various identifier-parsing productions now pre-parse an invalid module selector and convert it to unexpected syntax. In some cases this involves adjusting matching `can`/`at` methods to consume otherwise-invalid module selectors. • The previously-introduced `attach(_:to:)` mechanism is now used in more places. This makes all test cases inherited from the Swift tests pass, except for the `import` syntax which I’m a little iffy on.

Since some types and expressions can have module selectors, MissingTypeSyntax and MissingExprSyntax should have a module selector child and `attach(_:to:)` should be able to attach a module selector to them. This keeps the parser from erroring on both the module selector *and* the missing node.

Add tailored diagnostics for unexpected module selectors which offer either one or two fix-its: • Remove the module selector • Convert `Foo::bar` to `bar = Foo::bar` (in certain declaration syntaxes) Making these messages clear required adding `nameForDiagnostics` properties to a bunch of children, which also impacted other existing diagnostics (for the better, IMHO). If we don’t like those changes or think they need more work, this commit can be severed from the rest of the PR.

When `parseFunctionParameter()` parses two argument labels and then doesn’t find a type, it applies a heuristic to decide whether to reinterpret the second label as a type (and recover by inserting a colon between the labels). The code in this path could drop unexpected nodes between the two labels. Correct this issue and, if the unexpected syntax includes a module selector, reconstruct it and attach it to the type. This was probably a pre-existing bug, but the module selector tests managed to hit it through mutation testing.

The code assumed that dropping a `MissingTypeSyntax` wouldn’t lose any tokens.

beccadax · 2025-07-12T04:10:37Z

@swift-ci please test

nkcsgexi · 2025-07-14T18:41:10Z

@swift-ci please test macOS

nkcsgexi · 2025-07-14T18:41:24Z

@swift-ci please test Linux

nkcsgexi · 2025-07-14T21:06:15Z

This build failure seems to be real:

12:29:57  /Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/swift-syntax/Sources/SwiftSyntax/generated/SyntaxTraits.swift:209:7: error: protocol requirement 'moduleSelector' cannot be declared '@_spi' without a default implementation in a protocol extension
12:29:57  207 | 
12:29:57  208 |   @_spi(ExperimentalLanguageFeatures)
12:29:57  209 |   var moduleSelector: ModuleSelectorSyntax? {
12:29:57      |       `- error: protocol requirement 'moduleSelector' cannot be declared '@_spi' without a default implementation in a protocol extension
12:29:57  210 |     get
12:29:57  211 |     set

beccadax · 2025-07-15T00:09:10Z

I can't actually reproduce that failure locally, but I'm pushing a speculative fix.

beccadax · 2025-07-15T00:20:09Z

@swift-ci please test

beccadax · 2025-07-15T01:39:59Z

@swift-ci please test macOS

beccadax · 2025-07-15T01:40:08Z

@swift-ci please test Linux

beccadax · 2025-07-15T01:40:19Z

@swift-ci please test Windows

nkcsgexi · 2025-07-15T17:21:22Z

We are good now with macOS and Linux. Let's try Windows again. @swift-ci please test Windows.

nkcsgexi · 2025-07-15T21:54:33Z

hmm, the Windows CI hit a compiler crasher:

[519/528] Compiling SwiftDiagnostics Convenience.swift
error: compile command failed due to exception 3 (use -v to see invocation)
SIL memory lifetime failure in @$s11SwiftParser13TokenConsumerPAAE27consumeModuleSelectorTokens0C0Qz22moduleNameOrUnexpected_AF010colonColonC0SayAFG5extratSgyF: memory is initialized, but shouldn't be

rintaro

Apologies I didn't review this sooner. I still haven't look into the implementation closely, but here's the first round 🙇

rintaro · 2025-07-16T17:54:02Z

CodeGeneration/Sources/SyntaxSupport/CommonNodes.swift

+        documentation:
+          "A module selector. Some expressions can be prefixed with module selectors, so if one is parsed before an invalid expression, it will be inserted here.",
+        isOptional: true
+      ),


I don't feel using MissingExprSyntax as dangling module selector is the way to go. MissingExprSyntax is a "placeholder" for expression with unknown kind. But IMO, we can reasonably assume the user is to add an identifier after a module selector to form DeclReferenceExprSyntax.
I feel it's more natural to model it as DeclReferenceExprSyntax with missing baseName.

Same for MissingTypeSyntax

rintaro · 2025-07-16T18:35:28Z

Sources/SwiftParser/Names.swift

+  ///
+  /// - Precondition: `node` must, at minimum, have a descendant with an unexpected nodes child; it therefore cannot be
+  ///   a token or an empty collection.
+  func attach<Node: RawSyntaxNodeProtocol>(_ moduleSelector: RawModuleSelectorSyntax?, to node: Node) -> Node {


I've been thinking about this. but I'm still not convinced. Let's say for let Foo::Bar = <expr>, I don't think we can say the user tried to specify the module name. The user might mistyped let Foo: Bar = <expr>.

Whether developers might get confused between name declarations and name references is subjective, so I won’t argue that. But I don’t think it’s worth adding this extra complexity to the implementation. It looks overly complex. Also, rewriting parsed nodes leaves abandoned nodes in the arena, which is not great for memory usage. IMO, the module selector parsing should be mostly contained in parseDeclReferenceExpr(), (can)parseTypeIdentifier(), and (can)parseSimpleType() (for member types). Of course we should do our best to emit better diagnostics. But I don't feel this is the way. Also we can improve diagnostics later. Can we make it simple for the initial implementation?

rintaro · 2025-07-16T18:56:59Z

Sources/SwiftParser/Names.swift

+    // Technically the current token *should* be an identifier, but we also want to diagnose other tokens that might be
+    // used by accident (particularly keywords and `_`). However, we don't want to consume tokens which would make the
+    // surrounding structure mis-parse.
+    return self.at(anyIn: StructuralTokens.self) == nil


I don't think it's a good idea to (almost) always consider <token> :: as a module qualifier. E..g.

class ::

This looks to me an incomplete class declaration and just an orphan ::. Even if it's on the same line with no-space, I don't think we need to parse it as a module selector.

beccadax force-pushed the mod-squad branch from a20cd45 to d7511a3 Compare June 2, 2025 19:48

beccadax commented Jun 2, 2025

View reviewed changes

beccadax force-pushed the mod-squad branch from d7511a3 to 4c34e4b Compare June 4, 2025 00:57

[NFC] Thread experimental features through lexer

69d50cd

This capability is currently unused, but it’ll matter soon.

beccadax force-pushed the mod-squad branch 3 times, most recently from 1f47aa2 to eaa6956 Compare July 9, 2025 22:04

beccadax added 5 commits July 9, 2025 15:24

Add experimental colonColon token

1360c37

And make sure it doesn’t break Objective-C selector lexing.

[NFC] Refactor parsing of suppressed types

06f2097

[NFC] Tighten checking for PBD introducers

ee93aa6

Treating introducers as keywords is now always conditional on whether `shouldParsePatternBinding(introducer:)` returns `true`. That method has also been modified to correctly handle an edge case with wildcard patterns.

[NFC] Correct @_implements diagnostic typo

d848067

beccadax force-pushed the mod-squad branch 2 times, most recently from da6eff6 to f94b58a Compare July 10, 2025 03:00

beccadax force-pushed the mod-squad branch from f94b58a to 7652336 Compare July 10, 2025 21:17

beccadax marked this pull request as ready for review July 11, 2025 20:50

beccadax requested review from ahoppen, bnbarham and hamishknight as code owners July 11, 2025 20:50

beccadax added 7 commits July 11, 2025 16:26

[NFC] Refactor parseQualifiedTypeIdentifier()

0321c86

Changes it to share code with `parseTypeIdentifier()` and clean up the member type parsing a little. Also tweaks call sites of `parseTypeIdentifier()`.

Support module selectors in scoped imports

6a66de7

beccadax added 2 commits July 11, 2025 16:26

Don’t drop module selector in bad generic arg list

8adfdb1

The code assumed that dropping a `MissingTypeSyntax` wouldn’t lose any tokens.

beccadax force-pushed the mod-squad branch from 7652336 to 8adfdb1 Compare July 11, 2025 23:26

Fix issue with SPI protocol requirement

46eb9ee

beccadax requested a review from rintaro July 15, 2025 18:21

rintaro reviewed Jul 16, 2025

View reviewed changes

SwiftSyntax support for module selectors #3091

Are you sure you want to change the base?

SwiftSyntax support for module selectors #3091

Conversation

beccadax commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nkcsgexi commented Jun 2, 2025

Uh oh!

beccadax Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

rintaro Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beccadax Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

rintaro Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnbarham Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

beccadax commented Jul 10, 2025

Uh oh!

beccadax commented Jul 11, 2025

Uh oh!

beccadax commented Jul 11, 2025

Uh oh!

beccadax commented Jul 12, 2025

Uh oh!

nkcsgexi commented Jul 14, 2025

Uh oh!

nkcsgexi commented Jul 14, 2025

Uh oh!

nkcsgexi commented Jul 14, 2025

Uh oh!

beccadax commented Jul 15, 2025

Uh oh!

beccadax commented Jul 15, 2025

Uh oh!

beccadax commented Jul 15, 2025

Uh oh!

beccadax commented Jul 15, 2025

Uh oh!

beccadax commented Jul 15, 2025

Uh oh!

nkcsgexi commented Jul 15, 2025

Uh oh!

nkcsgexi commented Jul 15, 2025

Uh oh!

rintaro left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

beccadax commented Jun 2, 2025 •

edited

Loading

rintaro Jun 2, 2025 •

edited

Loading

rintaro Jul 16, 2025 •

edited

Loading

rintaro left a comment •

edited

Loading

rintaro Jul 16, 2025 •

edited

Loading

rintaro Jul 16, 2025 •

edited

Loading

rintaro Jul 16, 2025 •

edited

Loading