devops/kamal: Secure multi-host kamal deployments with private networking

[dmitry]

Overview

Document different approaches for securing multi-host Kamal deployments using private networking, focusing on practical implementations and real-world infrastructure choices.

This issue outlines a thorough exploration of secure multi-host deployments with Kamal, focusing on practical implementations and real-world scenarios. The modular approach allows teams to choose the networking solution that best fits their needs while maintaining security best practices.

Key Topics to Cover

1. Hetzner Cloud Implementation

• Using hCloud private networks (10.0.0.0/16)
• Integrating bare metal servers via vSwitch
• Example network architecture diagram
• Sample Kamal configuration for internal hostnames

2. Private Networking Options

A. WireGuard Approach

• Setting up WireGuard mesh between hosts
• Example configuration for 3+ hosts
• Advantages:
  • Modern, performant VPN
  • Relatively simple setup
  • Works across different providers
• Integration with Kamal configuration

B. Docker Swarm

• Using overlay networks
• Swarm initialization and node joining
• Benefits:
  • Native Docker integration
  • Built-in service discovery
  • Works well with Kamal's Docker-based architecture
• Example deployment architecture

C. Simple UFW Rules

• Basic UFW configuration template
• Allowing internal network traffic
• Cloudflare IP whitelist management
  • Script for auto-updating Cloudflare IPs
  • Cron job setup for keeping rules current

3. Example Architectures

Single Entry Point

                    │
                    ▼
             [Cloudflare]
                    │
                    ▼
            [Load Balancer]
                    │
        ┌──────────┴──────────┐
        ▼          ▼          ▼
  [App Server] [App Server] [App Server]
        │          │          │
        └──────────┼──────────┘
                   │
            [Internal Network]
                   │
        ┌─────────┴─────────┐
        ▼         ▼         ▼
    [Redis]   [Postgres]  [Cache]

Multi-Region Setup

• How to handle cross-region private networking
• Regional load balancers
• Data synchronization considerations

4. Implementation Guide

1. Network Setup

   • Private network creation
   • Firewall configuration
   • DNS configuration

2. Kamal Configuration

   • Using internal hostnames
   • Proxy/Caddy setup
   • Environment-specific configs

3. Security Hardening

   • Minimal port exposure
   • Certificate management
   • Network access controls

5. Real-world Examples

Hetzner Bare Metal + Cloud Hybrid

• vSwitch setup between dedicated and cloud servers
• Network performance considerations
• Cost optimization strategies

Multi-Cloud Setup

• WireGuard mesh across providers
• Load balancing considerations
• Failover strategies

Implementation Details Needed

• Network architecture diagrams
• Sample configurations
• Performance benchmarks
• Security best practices
• Monitoring recommendations
• Troubleshooting guide

Additional Ideas

1. Integration with HashiCorp Vault for secrets
2. Service mesh options (like Consul)
3. Network monitoring and debugging tools
4. Backup strategies across private networks
5. High availability configurations
6. Zero-trust network architecture examples

Questions to Address

1. How to handle network partitions?
2. Backup and restore procedures?
3. Scaling considerations?
4. Disaster recovery scenarios?
5. Cost implications of different approaches?

Resources to Include

• Links to relevant documentation
• Tool comparisons
• Community discussions
• Performance benchmarks
• Security advisories

This will serve as a comprehensive guide for teams looking to deploy Kamal in a secure, production environment with private networking.

TODO

• https://community.hetzner.com/tutorials/vpc-with-wireguard-pfsense
• https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch/