Incomplete fix for RemoteProtocolError - parallel requests cause unhandled NetworkError exceptions

[NickBorgers]

Summary

The fix in PR #88 for issue SpanPanel/span#156 is incomplete. While RemoteProtocolError is now handled correctly, parallel API requests in get_all_data() can still fail with unhandled exceptions when one request destroys the shared client while others are in-flight.

Related

• Original issue: Add retry logic for httpx RemoteProtocolError: Server disconnected without sending a response span#156
• Original fix: Recognize panel Keep-Alive at 5 sec, Handle httpx.RemoteProtocolError… #88

Problem

When get_all_data() runs parallel requests via asyncio.gather():

1. get_status() uses requires_auth=False → gets its own client (OK)
2. get_panel_state(), get_circuits(), get_storage_soe() use requires_auth=True → share self._client

When a stale connection causes RemoteProtocolError:

1. One task catches it and destroys self._client via __aexit__
2. Other parallel tasks still have in-flight requests using the same underlying httpx client
3. Those requests fail with ReadError, WriteError, or CloseError (all subclasses of NetworkError)
4. These exceptions are not caught by the retry loop
5. They bubble up and get wrapped as SpanPanelAPIError("Unexpected error: {e}")
6. Since these exceptions often have empty messages, logs show: "Unexpected error: "

Evidence

Home Assistant logs show two types of errors:

ERROR: Unexpected error: Server disconnected without sending a response.  ← RemoteProtocolError (handled)
ERROR: Unexpected error:                                                   ← ReadError/WriteError (unhandled, empty message)

The empty error messages are caused by httpx exceptions with no message:

>>> str(httpx.ReadError(''))
''
>>> str(httpx.WriteError(''))
''

Exception Hierarchy

httpx.TransportError
├── httpx.NetworkError
│   ├── httpx.ReadError      ← NOT HANDLED
│   ├── httpx.WriteError     ← NOT HANDLED
│   └── httpx.CloseError     ← NOT HANDLED
└── httpx.ProtocolError
    └── httpx.RemoteProtocolError  ← HANDLED in PR #88

Suggested Fix

1. Catch httpx.TransportError (or at minimum httpx.NetworkError) instead of just RemoteProtocolError:

except (httpx.RemoteProtocolError, httpx.NetworkError):
    # Server closed connection or network error - connection pool likely corrupted
    # Destroy client to force fresh connection pool on retry
    ...

2. Consider adding locking around client destruction/recreation to prevent race conditions, or

3. Use separate clients for parallel requests instead of sharing self._client

Environment

• span-panel-api: 1.1.14
• span (integration): 1.30b1 (manifest shows 1.3.0)
• Home Assistant: 2024.x
• Multiple SPAN panels (which may increase likelihood of hitting this race condition)

---

This issue was written by Claude (claude-opus-4-5-20250121) via Claude Code, based on analysis of the installed code and error logs.

🤖 Generated with Claude Code

[NickBorgers]

Minimal Reproducer

Here's a standalone script that demonstrates the race condition without requiring a real SPAN panel or credentials:

#!/usr/bin/env python3
"""
Demonstrates the race condition bug in span-panel-api v1.1.14

When parallel requests share a client and one encounters RemoteProtocolError,
the retry handler destroys the shared client. Other in-flight requests then
fail with ReadError/WriteError/CloseError which are NOT handled by the retry loop.

Usage:
    python3 span-race-condition-demo.py
"""

import asyncio
from contextlib import suppress
from typing import Optional

# Mock httpx exceptions if httpx is not installed
try:
    import httpx
except ImportError:
    class _MockHttpx:
        class RequestError(Exception): pass
        class TransportError(RequestError): pass
        class NetworkError(TransportError): pass
        class ReadError(NetworkError): pass
        class WriteError(NetworkError): pass
        class CloseError(NetworkError): pass
        class ProtocolError(TransportError): pass
        class RemoteProtocolError(ProtocolError): pass
        class AsyncClient:
            async def aclose(self): pass
    httpx = _MockHttpx()


class MockSpanClient:
    """Simplified version of SpanPanelClient that reproduces the bug."""

    def __init__(self):
        self._client = httpx.AsyncClient()
        self._retries = 3

    async def _retry_with_backoff(self, operation):
        """Retry logic from span-panel-api - only handles RemoteProtocolError"""
        max_attempts = self._retries + 1

        for attempt in range(max_attempts):
            try:
                return await operation()
            except httpx.RemoteProtocolError:
                print(f"  [Attempt {attempt + 1}] Caught RemoteProtocolError, destroying client...")
                if self._client is not None:
                    with suppress(Exception):
                        await self._client.aclose()
                    self._client = None
                if attempt < max_attempts - 1:
                    self._client = httpx.AsyncClient()
                    continue
                raise
            # BUG: NetworkError (ReadError, WriteError, CloseError) NOT caught!


async def demonstrate_race_condition():
    print("Scenario: Parallel requests with race condition")
    print("-" * 50)

    client = MockSpanClient()

    async def task_a():
        """Task A encounters RemoteProtocolError, destroys shared client"""
        print("  Task A: Starting...")
        await asyncio.sleep(0.1)
        print("  Task A: RemoteProtocolError! Destroying shared client...")
        if client._client is not None:
            await client._client.aclose()
            client._client = None
        raise httpx.RemoteProtocolError("Server disconnected")

    async def task_b():
        """Task B fails when client destroyed mid-request"""
        print("  Task B: Starting...")
        await asyncio.sleep(0.2)  # Still in-flight when A destroys client
        print("  Task B: Client destroyed mid-request!")
        raise httpx.ReadError("")  # Empty message

    async def task_c():
        print("  Task C: Starting...")
        await asyncio.sleep(0.2)
        print("  Task C: Client destroyed mid-request!")
        raise httpx.WriteError("")

    results = await asyncio.gather(task_a(), task_b(), task_c(), return_exceptions=True)

    print("\n  Results:")
    for i, (name, result) in enumerate(zip(["A", "B", "C"], results)):
        if isinstance(result, httpx.RemoteProtocolError):
            print(f"    Task {name}: RemoteProtocolError - would be retried ✓")
        elif isinstance(result, (httpx.ReadError, httpx.WriteError)):
            print(f"    Task {name}: {type(result).__name__}('{result}') - NOT handled ✗")


if __name__ == "__main__":
    asyncio.run(demonstrate_race_condition())

Output:

Scenario: Parallel requests with race condition
--------------------------------------------------
  Task A: Starting...
  Task B: Starting...
  Task C: Starting...
  Task A: RemoteProtocolError! Destroying shared client...
  Task B: Client destroyed mid-request!
  Task C: Client destroyed mid-request!

  Results:
    Task A: RemoteProtocolError - would be retried ✓
    Task B: ReadError('') - NOT handled ✗
    Task C: WriteError('') - NOT handled ✗

The empty ReadError('') and WriteError('') become SpanPanelAPIError("Unexpected error: ") in the actual code.

[NickBorgers]

Update: Progressive Degradation Over ~7 Hours

After the VM restart at 09:50, Home Assistant was stable with zero span_panel errors for approximately 7 hours. Then the errors returned and progressively worsened:

Timeline

• 09:50 - HA restarted, span_panel working correctly
• 16:41 - First errors appear (~7 hours later)
• 16:50 - HA UI becomes noticeably slow (8.5s HTTP response times)

Error Pattern When Degraded

ERROR: HTTP 412: Unexpected status code: 412 - "Access to protected HTTP endpoints require an API token"
ERROR: Request timed out after 30s
ERROR: Unexpected error:   ← (empty message - the NetworkError issue)

Impact on Home Assistant

• HTTP responses taking 8.5 seconds (normally <1s)
• UI login very slow
• WebSocket connections to home-automation service becoming lossy/inconsistent
• Other integrations timing out (Supervisor connections)

Hypothesis

The connection pool or authentication state appears to corrupt progressively over time, not immediately. This may explain why the fix "works" after a restart but degrades. Possible causes:

1. Slow memory/connection leak in httpx client
2. Auth token state becoming inconsistent after many retry cycles
3. Connection pool entries becoming stale but not being cleaned up

The ~7 hour timeframe might correlate with keep-alive timeouts accumulating or retry backoff state growing.

🤖 Generated with Claude Code

[NickBorgers]

Additional Detail: How span_panel Causes Home Assistant Unresponsiveness

The Mechanism

Home Assistant uses a single-threaded async event loop. When span_panel's retry loops block (30-second timeouts, connection recreation, exponential backoff), they starve all other tasks:

1. HTTP requests queue up - we observed 7-9 second response times for the root URL (normally <500ms)
2. WebSocket connections suffer - heartbeats miss deadlines, messages queue or drop
3. Other integrations time out - we saw "Timeout connecting to Supervisor" for multiple add-ons
4. UI becomes unusable - login takes 8+ seconds, dashboards don't load

Evidence from This Incident

# Normal HA response time
HTTP 200 in 0.3s

# During span_panel degradation  
HTTP 200 in 7.881704s
HTTP 200 in 6.203277s
HTTP 200 in 7.265911s
HTTP 200 in 9.619577s

The HTTP 200 response code masked the problem - HA was "up" but severely degraded. Our health check only looked for non-200 responses, so it didn't trigger a restart.

Temporary Mitigation

We updated our external health check to detect slow responses as failures:

# Now counts responses >5s as unhealthy
SLOW_THRESHOLD_SECONDS=5

# Combined with flapping detection:
# 3 slow responses in 10 minutes → automatic restart

This caught the degraded state and restarted HA, restoring normal operation. However, this is just a workaround - the span_panel integration will likely degrade again in ~7 hours.

Recommendation

The underlying issue needs to be fixed in span-panel-api. Until then, users with multiple SPAN panels may need external monitoring that can detect slow responses (not just failures) and restart HA proactively.

🤖 Generated with Claude Code