Return a single token for OpenQASM version statement from lexer #276
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit changes lexing the OpenQASM version statement. Flags recording failure to recognize major and minor versions are included. An example of the statement is ``` OPENQASM 3.1; ``` Here are some choices for lexing the openqasm version statement. This commit changes the implementaton from the first choice to the second. This was first implemented by doing nothing in the lexer to treat this statement. This is the way almost all keywords are lexed. All identifiers are recognized as keywords when parsing, rather than lexing. The number `3.1` is tokenized as a float literal (actually an integer literal, converted to float somewhere before parsing. This is inherited from r-a) But parsing does not have access to the input text, so this defers checking that `3.1` is a valid to the semantic analysis. This last step is easy enough to do, but was not done. Another way to implement lexing version statement is to recognize the entire statement as a token. This means recognizing the character sequence`OPENQASM` + whitespace + valid version number. If this fails, it will be lexed as an invalid identifier, or an invalid version statement, depending on where the error is. This allows catching errors where they ought to be caught, at the lexing and/or parsing stages. The version is then extracted from the token. This can be done with an api call created from the ungrammar, and/or coded by hand. Another way is to have the lexer recognize `OPENQASM` as a token, then a whitespace token, and finally a version number tokenkind, say `OPENQASM_VERSION`. This would be easier to parse and consume in semantic analysis. But the version number text would ordinarily be lexed as a float literal. Furthermore, the lexer produces a stream of tokens and maintains as little state a possible. We would need to break this abstraction in some way to accommodate this way of parsing the version string. For example, saving tokens and emitting all three when done. Or entering a special mode. The reference parser enters a "mode" enabled by the antlr parser generator. The lexer is currently very simple and fast. I don't want to experiment with adding complexity just for this purpose.
jlapeyre
force-pushed
the
change-lexing-openqasm-version
branch
from
64a090a to
bbcfa1d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit changes lexing the OpenQASM version statement. Flags recording failure to recognize major and minor versions are included.
An example of the statement is
Here are some choices for lexing the openqasm version statement. This commit changes the implementaton from the first choice to the second.
This was first implemented by doing nothing in the lexer to treat this statement. This is the way almost all keywords are lexed. All identifiers are recognized as keywords when parsing, rather than lexing. The number
3.1is tokenized as a float literal (actually an integer literal, converted to float somewhere before parsing. This is inherited from r-a) But parsing does not have access to the input text, so this defers checking that3.1is a valid to the semantic analysis. This last step is easy enough to do, but was not done.Another way to implement lexing version statement is to recognize the entire statement as a token. This means recognizing the character sequence
OPENQASM+ whitespace + valid version number. If this fails, it will be lexed as an invalid identifier, or an invalid version statement, depending on where the error is. This allows catching errors where they ought to be caught, at the lexing and/or parsing stages. The version is then extracted from the token. This can be done with an api call created from the ungrammar, and/or coded by hand.Another way is to have the lexer recognize
OPENQASMas a token, then a whitespace token, and finally a version number tokenkind, sayOPENQASM_VERSION. This would be easier to parse and consume in semantic analysis. But the version number text would ordinarily be lexed as a float literal. Furthermore, the lexer produces a stream of tokens and maintains as little state a possible. We would need to break this invariant in some way to accommodate this way of parsing the version string. For example, saving tokens and emitting all three when done. Or entering a special mode. The reference parser enters a "mode" enabled by the antlr parser generator. The lexer is currently very simple and fast. I don't want to experiment with adding complexity just for this purpose.