Agent: Detect RESOURCE_EXHAUSTED when parsing LLM response and dont

[pokryfka]

Task

Detect RESOURCE_EXHAUSTED when parsing LLM response and dont try to retry

Changes

• src/agent/loop.py
• src/llm/base.py
• tests/test_executor.py

model: gemini-2.5-pro
thread_id: 82e2907280d61640

Summary by CodeRabbit

• New Features

  • Added graceful error handling for LLM resource exhaustion with user-friendly error messages when the system runs out of resources.
  • Enhanced JSON parser to explicitly detect and signal resource exhaustion conditions.

• Tests

  • Added test coverage for resource exhaustion error handling.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds handling for LLM resource exhaustion errors across the agent and parser layers. A new ResourceExhaustedError exception is introduced to detect "RESOURCE_EXHAUSTED" responses, caught and logged in the agent loop, with accompanying test coverage.

Changes

Cohort / File(s)	Summary
Error Handling Implementation src/llm/base.py, src/agent/loop.py	Introduces new ResourceExhaustedError exception; parser detects "RESOURCE_EXHAUSTED" in LLM responses and raises the error; agent loop wraps graph.ainvoke in try/except to catch and log exhaustion with user-facing abort message. Minor regex pattern adjustment in brace-based JSON extraction.
Test Coverage tests/test_executor.py	Adds test test_robust_json_parser_resource_exhausted to verify RobustJsonOutputParser.parse raises ResourceExhaustedError when input contains "RESOURCE_EXHAUSTED".

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 With whiskers twitched and eyes so bright,
We catch the "exhausted" in the night,
A new exception hops along,
To handle when resources gone,
Error paths so clean and right! ✨

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title is incomplete and appears to be cut off mid-sentence ('and dont' lacks proper grammar and continuation), making it unclear what action should not be taken.	Complete the title with proper grammar and clarity, e.g., 'Agent: Detect RESOURCE_EXHAUSTED when parsing LLM response and don't retry' or similar.
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch agent/detect-resource_exhausted-when-parsing-l

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (4)

tests/test_executor.py (3)

1-8: Missing from __future__ import annotations.

Per coding guidelines, every Python module should include this import.

♻️ Add future annotations import

 """Tests for the command executor."""
 
+from __future__ import annotations
+
 from pathlib import Path
 
 import pytest

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_executor.py` around lines 1 - 8, This module is missing the
required top-level future annotations import; add "from __future__ import
annotations" as the very first import in the file so type annotations for
functions/classes (e.g., test imports referencing list_files, read_file,
run_command, ResourceExhaustedError, RobustJsonOutputParser) use postponed
evaluation; ensure it appears before any other imports or module-level
docstring.

---

8-8: Consider test file organization.

This test file is named test_executor.py but now includes tests for RobustJsonOutputParser from src/llm/base.py. Consider either:

1. Creating a separate test_llm_base.py or test_parser.py for parser tests, or
2. Renaming/restructuring to reflect the broader scope.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_executor.py` at line 8, The tests currently in
tests/test_executor.py import and exercise RobustJsonOutputParser and
ResourceExhaustedError from src/llm/base.py, which misplaces parser-specific
tests in an executor-focused file; move all tests that reference
RobustJsonOutputParser (and any parser-related helpers) into a new test file
such as tests/test_llm_base.py or tests/test_parser.py (or rename the current
file to reflect the broader scope), update imports there to import
RobustJsonOutputParser and ResourceExhaustedError, and leave only
executor-related tests in tests/test_executor.py to keep test responsibilities
aligned with the file name.

---

48-53: Test correctly verifies the exception is raised.

The test validates the basic scenario. Consider adding a test case to verify that valid JSON containing the error string still raises the exception (to confirm the check happens before JSON parsing):

def test_robust_json_parser_resource_exhausted_in_json():
    """Verify exception raised even if error text is embedded in valid JSON."""
    parser = RobustJsonOutputParser()
    json_with_error = '{"error": "RESOURCE_EXHAUSTED", "code": 429}'
    with pytest.raises(ResourceExhaustedError):
        parser.parse(json_with_error)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/test_executor.py` around lines 48 - 53, Add a new unit test that
verifies RobustJsonOutputParser.parse raises ResourceExhaustedError even when
the error token appears inside syntactically valid JSON: create a function
(e.g., test_robust_json_parser_resource_exhausted_in_json) that instantiates
RobustJsonOutputParser, sets json_with_error = '{"error": "RESOURCE_EXHAUSTED",
"code": 429}', and asserts with pytest.raises(ResourceExhaustedError) that
parser.parse(json_with_error) is called; this ensures the error-string check
runs before JSON decoding.

src/llm/base.py (1)

40-41: Consider case-insensitive matching for robustness.

The current check is case-sensitive. If LLM providers might return variations like "resource_exhausted" or "Resource_Exhausted", consider making the check case-insensitive.

♻️ Optional: Case-insensitive check

-        if "RESOURCE_EXHAUSTED" in text:
+        if "RESOURCE_EXHAUSTED" in text.upper():
             raise ResourceExhaustedError("LLM response indicates resource exhaustion")

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/llm/base.py` around lines 40 - 41, The check for "RESOURCE_EXHAUSTED" is
case-sensitive; update the condition that inspects the variable text (in
src/llm/base.py) to perform a case-insensitive match (e.g., compare against
text.lower() or text.casefold(), or use a regex with IGNORECASE) before raising
ResourceExhaustedError so variants like "resource_exhausted" or
"Resource_Exhausted" are detected reliably.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/llm/base.py`:
- Around line 40-41: The check for "RESOURCE_EXHAUSTED" is case-sensitive;
update the condition that inspects the variable text (in src/llm/base.py) to
perform a case-insensitive match (e.g., compare against text.lower() or
text.casefold(), or use a regex with IGNORECASE) before raising
ResourceExhaustedError so variants like "resource_exhausted" or
"Resource_Exhausted" are detected reliably.

In `@tests/test_executor.py`:
- Around line 1-8: This module is missing the required top-level future
annotations import; add "from __future__ import annotations" as the very first
import in the file so type annotations for functions/classes (e.g., test imports
referencing list_files, read_file, run_command, ResourceExhaustedError,
RobustJsonOutputParser) use postponed evaluation; ensure it appears before any
other imports or module-level docstring.
- Line 8: The tests currently in tests/test_executor.py import and exercise
RobustJsonOutputParser and ResourceExhaustedError from src/llm/base.py, which
misplaces parser-specific tests in an executor-focused file; move all tests that
reference RobustJsonOutputParser (and any parser-related helpers) into a new
test file such as tests/test_llm_base.py or tests/test_parser.py (or rename the
current file to reflect the broader scope), update imports there to import
RobustJsonOutputParser and ResourceExhaustedError, and leave only
executor-related tests in tests/test_executor.py to keep test responsibilities
aligned with the file name.
- Around line 48-53: Add a new unit test that verifies
RobustJsonOutputParser.parse raises ResourceExhaustedError even when the error
token appears inside syntactically valid JSON: create a function (e.g.,
test_robust_json_parser_resource_exhausted_in_json) that instantiates
RobustJsonOutputParser, sets json_with_error = '{"error": "RESOURCE_EXHAUSTED",
"code": 429}', and asserts with pytest.raises(ResourceExhaustedError) that
parser.parse(json_with_error) is called; this ensures the error-string check
runs before JSON decoding.

---

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 800c792 and 05003aa.

📒 Files selected for processing (3)

• src/agent/loop.py
• src/llm/base.py
• tests/test_executor.py