Fixed panic mode error recovery logics

[ehwan]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @ehwan, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on a comprehensive overhaul of the panic mode error recovery system within the parsing library. By introducing a TriState enum and integrating it into the grammar's state management, the changes enable more intelligent and robust error handling. The modifications span from the core grammar builder to the parser's runtime context, ensuring that the system can gracefully recover from parsing errors in a more predictable manner.

Highlights

• Enhanced Error Recovery Logic: The core error recovery mechanism, often referred to as 'panic mode,' has been significantly improved for both deterministic and non-deterministic parsers. This involves a more sophisticated approach to identifying states capable of handling error tokens and managing the parser's internal stacks during recovery.
• Introduction of TriState Enum: A new TriState enum (False, Maybe, True) has been introduced to provide a more nuanced representation of boolean logic, particularly for indicating whether a parser state can accept an error token. This allows for more granular control and better decision-making during error handling.
• Type Parameter Refinement: The Grammar struct and many of its associated methods now consistently use TerminalSymbol<Term> instead of a raw Term type parameter. This change standardizes the representation of terminal symbols throughout the grammar building process, improving type safety and clarity.
• Parser State Metadata for Error Handling: Parser states now include a can_accept_error: TriState field. This field is computed during grammar construction and propagated through state merging, explicitly marking states that are designed to handle error tokens. This metadata is then utilized by the parser's error recovery routines.
• Generated Code Updates: The code generation process has been updated to emit the new can_accept_error field within the IntermediateState definitions, ensuring that generated parsers correctly incorporate the enhanced error recovery capabilities.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a more sophisticated error recovery mechanism using a TriState to optimize finding a recovery state. The changes are extensive, refactoring terminal symbol handling to use TerminalSymbol<Term> and updating parser state generation and merging logic. While the overall approach is sound, I've found a critical bug in the BitOr implementation for TriState which incorrectly handles merging states' error recovery capabilities. This could lead to incorrect or inefficient error recovery.

[gemini-code-assist]

The bitor implementation for TriState seems incorrect. When merging two states, if one of them can definitely accept an error (True), the merged state should also be True. The current logic can result in Maybe in cases like True | False, which is incorrect. This can lead to less efficient error recovery, as it might trigger expensive checks unnecessarily, and could potentially lead to recovery failures.

The logic should prioritize True over Maybe, and Maybe over False. Here is a corrected implementation:

        match (self, rhs) {
            (TriState::True, _) | (_, TriState::True) => TriState::True,
            (TriState::Maybe, _) | (_, TriState::Maybe) => TriState::Maybe,
            (TriState::False, TriState::False) => TriState::False,
        }