-
-
Notifications
You must be signed in to change notification settings - Fork 253
Description
This is my first bug report, so bare with me. I tried everything possible to get it to work and also used Claude for debugging this but to no avail. Only option was to downgrade to v1.18.4 (same version as my working VPS periphery instance). The bug report was also generated with Claude as it has all the needed information (hopefully) and since it is my first report, I wanted to give you all the needed details.
Komodo Periphery v1.19.5 - Agent Crashes on Container Management API Requests
Summary
Komodo Periphery v1.19.5 crashes/exits when receiving container management API requests from Komodo Core, preventing users from viewing containers, restarting containers, or deploying stacks. The issue does not occur in v1.18.4. Both versions are pulled with the latest tag, indicating a regression in the latest release.
Environment Details
VPS (Working - v1.18.4)
- OS: Debian GNU/Linux 12 (bookworm)
- Kernel: 6.1.0-41-amd64
- Docker: 29.1.3
- Storage Driver: overlay2
- Image Hash:
sha256:0a3ccd9294b13d3c5942d1e20392c81e28369ba1aaba6b17bfd5b35907760b98 - Komodo Periphery Version: v1.18.4
LXC Container (Broken - v1.19.5)
- Hypervisor: Proxmox VE 9.1.2
- Container OS: Ubuntu 24.04.3 LTS
- Kernel: 6.17.2-2-pve x86_64
- Docker: 29.1.3 (installed via official Docker documentation)
- Storage Driver: overlay2
- Komodo Periphery Version: v1.19.5
Komodo Core
- OS: Ubuntu 24.04.3 LTS
- Kernel: 6.8.0-90-generic x86_64
- Version: v1.17.5
Issue Description
Both VPS and LXC use identical docker-compose and .env files, identical passkeys, and were both pulling from the latest tag. However:
- VPS pulled v1.18.4 and works perfectly
- LXC pulled v1.19.5 and crashes on container management operations
Steps to Reproduce
- Deploy Komodo Periphery v1.19.5 on an LXC container using standard docker-compose setup
- Register the agent with Komodo Core
- In Core UI, navigate to agent's "Containers" tab
- Attempt to view container list or execute container management commands (restart, deploy)
Expected Behavior
- Core UI should display list of running containers
- Container restart and deployment operations should succeed
- Agent should remain running and responsive to API requests
Actual Behavior
- Core receives TLS error:
peer closed connection without sending TLS close_notify - Agent logs show:
Exiting all active Terminals for shutdown - Agent container restarts due to
restart: unless-stoppedpolicy - Container operations fail with "failed to restart all containers on host" error
- Fire-and-forget commands (like
docker system prune) execute successfully - System shows agent as "OK" but container queries return no results
Root Cause Analysis
The issue appears to be introduced between v1.18.4 and v1.19.5. The agent successfully:
- Authenticates with Core (health check passes)
- Receives API requests (shutdown log shows it's triggered)
- Executes fire-and-forget Docker commands (system prune works)
But fails to:
- Query Docker for container information
- Return API responses to Core
- Maintain connection during container operations
This suggests a regression in response handling, serialization, or connection management in v1.19.5.
Workaround
Using the exact image hash from the VPS (v1.18.4) resolves all issues completely:
image: ghcr.io/moghtech/komodo-periphery@sha256:0a3ccd9294b13d3c5942d1e20392c81e28369ba1aaba6b17bfd5b35907760b98With this hash, the LXC container exhibits identical behavior to the VPS and all operations succeed.
Core Error Log (v1.19.5)
2025-12-23T19:51:08.751452Z WARN core::api::execute: /execute request error:
failed to restart all containers on host: failed at request to periphery:
error sending request for url (https://10.25.1.101:8120/):
client error (SendRequest): connection error:
peer closed connection without sending TLS close_notify:
https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof
Agent Log During Container Query (v1.19.5)
2025-12-23T19:51:08.646384Z INFO periphery: Exiting all active Terminals for shutdown
INFO: No config paths found, using default config
2025-12-23T19:51:08.956568Z INFO periphery: Komodo Periphery version: v1.19.5
Configuration Files
Both VPS and LXC use identical setup:
docker-compose.yml
services:
komodo-periphery:
image: ghcr.io/moghtech/komodo-periphery:latest
container_name: komodo-periphery
restart: unless-stopped
network_mode: host
env_file:
- ./periphery.env
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${PERIPHERY_ROOT_DIRECTORY:-/etc/komodo}:${PERIPHERY_ROOT_DIRECTORY:-/etc/komodo}
- /:/host:roperiphery.env
PERIPHERY_BIND_IP=0.0.0.0
PERIPHERY_PORT=8120
PERIPHERY_PASSKEYS=<shared-with-core>
PERIPHERY_ROOT_DIRECTORY=/opt/komodo
PERIPHERY_SSL_ENABLED=true
Impact
- Users cannot manage containers on agents using v1.19.5
- Deployments fail on v1.19.5 agents
- Read-only operations (system prune) work but container management is completely broken
- Automatic
latesttag pulls may break existing setups unexpectedly
Request
Please investigate the regression between v1.18.4 and v1.19.5, particularly around:
- Container query response handling
- Connection management during API responses
- TLS close sequence in response flows