BrsFile ast toString()

[TwitchBronBron]

Adds the ability to call .toString() on any BrightScript/BrighterScript AST node and get the source code it represents, including all leading trivia (whitespace, comments, newlines, colons). For an unmodified AST, ast.toString() returns exactly the text that was parsed.

This PR has been rebuilt on top of v1, which already tracks trivia on every token, so most of the original v0 work (lexer trivia, leadingWhitespace) is no longer needed.

const parser = Parser.parse(text);
parser.ast.toString() === text; // true

// works on any node, and reflects changes made to the AST
assignment.tokens.name.text = 'firstName';
parser.ast.toString(); // `firstName = "bob"`

How it works

• AstNode has a new abstract toSourceNode(state: TranspileState): SourceNode, implemented by every statement and expression. toString() calls it with a blank TranspileState. Because it returns a SourceNode, the output carries source-map positions.
• New TranspileState helpers:
  • tokenToSourceNodeWithTrivia
  • nodeToSourceNode (also writes any annotations)
  • nodesToSourceNode (writes a list of nodes with their separators)
  • statementsToSourceNode

Parser changes

Some tokens were consumed by the parser but never stored, so they are now kept on the AST:

• tokens.commas on CallExpression, FunctionExpression, ArrayLiteralExpression, IndexedGetExpression, IndexedSetStatement, DimStatement, CallfuncExpression, InlineInterfaceExpression, TypedFunctionTypeExpression, and InterfaceMethodStatement. commas[i] is the comma after item i.
• tokens.expressionBegins and tokens.expressionEnds (the ${ and } tokens) on template strings.
• tokens.dot for the obj.[index] syntax on IndexedGetExpression and IndexedSetStatement.
• tokens.eof on the root Body. It holds everything after the last statement.
• Colons that are part of the AST (AA key: value, ternary a ? b : c, label:) are removed from the next token's leading trivia, so they aren't written twice.
• When a file has syntax errors, any token the parser skipped is moved into the leading trivia of the next token in the AST. This means even broken code round-trips, which matters for the language server. The extra pass only runs when the parse produced diagnostics, so valid files pay nothing for it.

Plugin-created nodes

Nodes and tokens created by plugins usually have no location and no trivia. Written as-is, they would run into their neighbors (x=5+3, statements on the same line). So when a token or node has no location and no leading trivia, toString() adds sensible defaults:

• spaces around operators, keywords, = and as
• , between arguments
• statements start on a new line

Tokens and nodes that came from the parser are never changed.

Bugs fixed along the way

• TypedFunctionTypeExpression had its left and right parens swapped.
• Cloning an AAMemberExpression dropped its comma, and cloning an AnnotationExpression dropped its call arguments.
• exitwhile lost its leading trivia when it was split into exit + while.
• The parser looped forever when #error was the last line of a file with no trailing newline.
• The lexer included an unexpected character in the text of the next token, e.g. a newline token with text "%\n".
• Characters the lexer doesn't recognize (e.g. a stray |) are now kept as TokenKind.UnexpectedCharacter leading trivia instead of being dropped, so they round-trip. They are never added to the token list, so parsing, diagnostics, and transpile output are unchanged.
• When end sub or end class was missing at the end of a file, the previous token was stored a second time as the end token.
• createToken(TokenKind.ForEach), createToken(TokenKind.ExitWhile), and createToken(TokenKind.At) now default to for each, exit while, and @.
• A function with a sub/function token but no end token was transpiled (and typedef'd) as end [object Object].
• TranspileState.sourceNode() crashed when given an undefined locatable.
• ExitStatement.transpile crashed without a loop type token, and plugin-created exit/continue loop types were not separated by a space.
• InterfaceMethodStatement.leadingTrivia crashed when the function type token was missing.
• Typedefs dropped comments above an annotation (they're part of the annotation's @ token trivia), and moved comments between an annotation and its statement above the annotation. A new BrsTranspileState.getTypedefLeadingCommentsAndAnnotations writes both in source order for every typedef.
• MethodStatement.leadingTrivia skipped the public/override modifiers, so comments above those methods were dropped from typedefs and from the transpiled method function.

Verification

• The full test suite passes, and new specs cover every node type, clones, plugin-created nodes, the new parser tokens, and syntax-error recovery.
• Coverage: every line and branch added by this PR is covered (100% of statements and branches in the changed source files). AstOutput.spec.ts also covers previously untested paths in the existing transpile and getTypedef methods, including SOURCE_NAMESPACE_NAME and SOURCE_NAMESPACE_ROOT_NAME, typedef leading comments and annotations, and nodes missing optional parts. The only uncovered branches left in those methods are defensive fallbacks that can't be reached (for example getLeadingComments(...) ?? [], since it always returns an array).
• Every parse performed by the full test suite was checked with ast.toString() === input, and every unique input matches exactly.
• 2604 real-world .brs/.bs files (rooibos, the Roku SDK samples, and a large production app) all round-trip exactly.
• Transpiled output for 1365 real-world files was compared before and after this change. Ignoring comments and blank lines, the code is identical in every file. The only difference is 496 comment lines that used to be dropped and are now kept, mostly doc comments above public/override methods.

Known limitations

• Exact fidelity requires locations, which is the default (trackLocations: true). With trackLocations: false, every token looks like it was created by a plugin, so default spacing may be added. trackLocations: false is only used in tests.

🤖 Generated with Claude Code