CST: Serialize and print EOF trivia properly #247
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JohnnyMorganz
commented
May 2, 2025
| return { | ||
| parse = parse, | ||
| parseexpr = parseexpr, | ||
| parsefile = parsefile, |
There was a problem hiding this comment.
Not 100% on this API name. It was unclear if this should've just replaced "parse", or if we still wanted the ability to parse a block independently of whether it was a whole file. Happy to change it if there's suggestions, or can do it in a followup when we standardise the whole API after it's complete.
There was a problem hiding this comment.
I think this is a fine name for now, i suppose
There was a problem hiding this comment.
cool, will leave it alone for now, and revisit it afterwards
aatxe
approved these changes
May 4, 2025
green-real
pushed a commit
to green-real/lute
that referenced
this pull request
May 19, 2025
The CST works by attaching trivia between tokens as trailing trivia of the previous token or leading trivia of the currently processed token. However, right now the CST does not serialize any trailing trivia of the file properly (e.g. whitespace such as newline at EOF, or files composed solely of comments). This is because we do not process a final token to attach leading trivia to. This PR introduces a final "EOF" token that is composed solely of leading trivia at the end of the file. This is not a concept in the Luau AST itself, so we need to extend our parse result to carry the "eof" token separately, independent of the AstStatBlock. An alternative approach could be to attach all trivia at the end of the file as trailing trivia of the last processed token. However, this ends up special casing this particular token and breaks the guarantee that the trailing trivia will only consist of trivia up to and including a single newline character. As such, downstream tooling would potentially need to handle final statement / expressions in files separately to all other statements / expressions if they cared about trivia processing (e.g., formatters). Also, this would not work for files consisting solely of trivia, as there is no token present.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The CST works by attaching trivia between tokens as trailing trivia of the previous token or leading trivia of the currently processed token. However, right now the CST does not serialize any trailing trivia of the file properly (e.g. whitespace such as newline at EOF, or files composed solely of comments). This is because we do not process a final token to attach leading trivia to.
This PR introduces a final "EOF" token that is composed solely of leading trivia at the end of the file. This is not a concept in the Luau AST itself, so we need to extend our parse result to carry the "eof" token separately, independent of the AstStatBlock.
An alternative approach could be to attach all trivia at the end of the file as trailing trivia of the last processed token. However, this ends up special casing this particular token and breaks the guarantee that the trailing trivia will only consist of trivia up to and including a single newline character. As such, downstream tooling would potentially need to handle final statement / expressions in files separately to all other statements / expressions if they cared about trivia processing (e.g., formatters). Also, this would not work for files consisting solely of trivia, as there is no token present.