Separate maintenance mode (503) from infrastructure/application errors

examples/nginx/README.md

@@ -0,0 +1,212 @@
+# NGINX Configuration Examples for Error Handling
+
+This directory contains example NGINX configuration files that demonstrate proper error handling patterns, specifically addressing the issue where HTTP 503 was being used for both maintenance mode and other server errors.
+
+## Problem Statement
+
+Previously, the maintenance page was being served for many different errors because HTTP 503 (Service Unavailable) was being used to indicate maintenance mode, but 503 was also being generated for other reasons like backend failures, timeouts, and other issues.
+
+## Solution
+
+This configuration separates maintenance mode from other error conditions:
+
+### Error Code Usage
+
+- **503 (Service Unavailable)**: Reserved ONLY for scheduled maintenance mode
+  - Triggered by the presence of `/var/www/maintenance.flag` file
+  - Displays `maintenance.html`
+
+- **502 (Bad Gateway)**: Backend server is down or unreachable
+  - Occurs when upstream server cannot be reached
+  - Displays `502.html`
+
+- **504 (Gateway Timeout)**: Backend server timeout
+  - Occurs when upstream server takes too long to respond
+  - Displays `504.html`
+
+- **500 (Internal Server Error)**: Application errors
+  - Occurs when the application encounters an error
+  - Displays `500.html`
+
+- **429 (Too Many Requests)**: Rate limiting
+  - Occurs when rate limits are exceeded
+  - Displays `429.html`
+
+## Files
+
+```
+examples/nginx/
+├── nginx.conf                  # Server block configuration (for inclusion)
+├── nginx-standalone.conf       # Complete standalone configuration (for testing)
+├── test-config.sh             # Test script to validate configuration
+├── error-pages/
+│   ├── maintenance.html        # Scheduled maintenance page (503)
+│   ├── 502.html               # Bad Gateway error page
+│   ├── 504.html               # Gateway Timeout error page
+│   ├── 500.html               # Internal Server Error page
+│   └── 429.html               # Too Many Requests page
+└── README.md                   # This file
+```
+
+## Usage
+
+### Basic Setup
+
+**Option 1: Include in existing configuration (recommended)**
+
+1. Copy the `nginx.conf` snippet to your NGINX sites directory:
+   ```bash
+   sudo cp nginx.conf /etc/nginx/sites-available/your-site
+   sudo ln -s /etc/nginx/sites-available/your-site /etc/nginx/sites-enabled/
+   ```
+
+2. Copy the error pages to your web root:
+   ```bash
+   sudo cp -r error-pages/* /var/www/html/
+   ```
+
+3. Test and reload NGINX:
+   ```bash
+   sudo nginx -t
+   sudo systemctl reload nginx
+   ```
+
+**Option 2: Standalone configuration (for testing)**
+
+1. Use the `nginx-standalone.conf` for a complete configuration:
+   ```bash
+   sudo cp nginx-standalone.conf /etc/nginx/nginx.conf
+   sudo cp -r error-pages/* /var/www/html/
+   sudo nginx -t
+   sudo systemctl restart nginx
+   ```
+
+### Enabling Maintenance Mode
+
+To enable maintenance mode, simply create the maintenance flag file:
+
+```bash
+sudo touch /var/www/maintenance.flag
+```
+
+To disable maintenance mode, remove the flag file:
+
+```bash
+sudo rm /var/www/maintenance.flag
+```
+
+### Customization
+
+#### Change Maintenance Flag Location
+
+Edit the `nginx.conf` file and update the path in the `if` statement:
+
+```nginx
+if (-f /your/custom/path/maintenance.flag) {
+    return 503;
+}
+```
+
+#### Customize Error Pages
+
+All error pages are standard HTML files. You can edit them to match your branding:
+
+```bash
+sudo nano /var/www/html/maintenance.html
+```
+
+#### Add Backend Servers
+
+Update the `upstream backend` block in `nginx.conf`:
+
+```nginx
+upstream backend {
+    server backend1.example.com:8080 max_fails=3 fail_timeout=30s;
+    server backend2.example.com:8080 max_fails=3 fail_timeout=30s;
+    keepalive 32;
+}
+```
+
+## Testing
+
+### Automated Testing
+
+Run the provided test script to validate the configuration:
+
+```bash
+cd examples/nginx
+./test-config.sh
+```
+
+The test script validates:
+- NGINX configuration syntax
+- Presence of all error page files
+- HTML structure of error pages
+- Key configuration features
+- Uniqueness of error page content
+
+### Test Maintenance Mode
+
+```bash
+# Enable maintenance
+sudo touch /var/www/maintenance.flag
+curl -I http://localhost
+# Should return HTTP 503 with maintenance page
+
+# Disable maintenance
+sudo rm /var/www/maintenance.flag
+curl -I http://localhost
+# Should return normal response
+```
+
+### Test Other Error Pages
+
+You can test other error pages by temporarily configuring NGINX to return specific status codes:
+
+```nginx
+location /test-502 {
+    return 502;
+}
+
+location /test-504 {
+    return 504;
+}
+```
+
+## Integration with Tracing-Stackdriver
+
+When using this NGINX configuration with applications that use the `tracing-stackdriver` library, error responses will be properly logged with their correct HTTP status codes, making it easier to distinguish between:
+
+- Scheduled maintenance (503)
+- Infrastructure issues (502, 504)
+- Application errors (500)
+- Rate limiting (429)
+
+This improves observability and helps with debugging and monitoring in Google Cloud Operations Suite.
+
+## Best Practices
+
+1. **Always use the maintenance flag file**: Never return 503 directly for other error conditions
+2. **Monitor upstream health**: Configure proper health checks in your upstream blocks
+3. **Set appropriate timeouts**: Adjust `proxy_*_timeout` values based on your application's needs
+4. **Log errors appropriately**: Ensure error logs are being sent to your monitoring system
+5. **Test maintenance mode regularly**: Verify that maintenance mode works as expected before you need it
+6. **Keep error pages simple**: Error pages should load quickly and have minimal dependencies
+7. **Provide helpful information**: Include relevant contact information or status page links
+
+## Troubleshooting
+
+### Maintenance page not showing
+- Check that `/var/www/maintenance.flag` exists
+- Verify NGINX configuration syntax: `sudo nginx -t`
+- Check NGINX error logs: `sudo tail -f /var/log/nginx/error.log`
+
+### Wrong error page displayed
+- Verify error page files exist in `/var/www/html/`
+- Check file permissions: `sudo chmod 644 /var/www/html/*.html`
+- Ensure `error_page` directives are not being overridden in other config files
+
+### Backend connection issues
+- Verify upstream server is running
+- Check firewall rules between NGINX and backend
+- Test direct connection: `curl http://backend-ip:port`

examples/nginx/error-pages/429.html

@@ -0,0 +1,57 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Too Many Requests</title>
+    <style>
+        body {
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
+            background: linear-gradient(135deg, #feca57 0%, #ff9ff3 100%);
+            color: #333;
+            display: flex;
+            justify-content: center;
+            align-items: center;
+            min-height: 100vh;
+            margin: 0;
+            padding: 20px;
+        }
+        .container {
+            background: white;
+            border-radius: 10px;
+            box-shadow: 0 10px 40px rgba(0,0,0,0.1);
+            padding: 40px;
+            max-width: 600px;
+            text-align: center;
+        }
+        h1 {
+            color: #ee5a6f;
+            margin-top: 0;
+            font-size: 2.5em;
+        }
+        p {
+            color: #666;
+            line-height: 1.6;
+            font-size: 1.1em;
+        }
+        .icon {
+            font-size: 4em;
+            margin-bottom: 20px;
+        }
+        .status-code {
+            color: #999;
+            font-size: 0.9em;
+            margin-top: 30px;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <div class="icon">🚦</div>
+        <h1>Too Many Requests</h1>
+        <p>You've made too many requests in a short period.</p>
+        <p>Please wait a moment before trying again.</p>
+        <div class="status-code">HTTP 429 - Too Many Requests</div>
+    </div>
+</body>
+</html>

examples/nginx/error-pages/500.html

@@ -0,0 +1,76 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Internal Server Error</title>
+    <style>
+        body {
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
+            background: linear-gradient(135deg, #ff6b6b 0%, #c92a2a 100%);
+            color: #333;
+            display: flex;
+            justify-content: center;
+            align-items: center;
+            min-height: 100vh;
+            margin: 0;
+            padding: 20px;
+        }
+        .container {
+            background: white;
+            border-radius: 10px;
+            box-shadow: 0 10px 40px rgba(0,0,0,0.1);
+            padding: 40px;
+            max-width: 600px;
+            text-align: center;
+        }
+        h1 {
+            color: #c92a2a;
+            margin-top: 0;
+            font-size: 2.5em;
+        }
+        p {
+            color: #666;
+            line-height: 1.6;
+            font-size: 1.1em;
+        }
+        .icon {
+            font-size: 4em;
+            margin-bottom: 20px;
+        }
+        .status-code {
+            color: #999;
+            font-size: 0.9em;
+            margin-top: 30px;
+        }
+        .action {
+            margin-top: 20px;
+        }
+        button {
+            background: #c92a2a;
+            color: white;
+            border: none;
+            padding: 12px 24px;
+            border-radius: 5px;
+            font-size: 1em;
+            cursor: pointer;
+            transition: background 0.3s;
+        }
+        button:hover {
+            background: #a61e1e;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <div class="icon">⚠️</div>
+        <h1>Internal Server Error</h1>
+        <p>Something went wrong on our end.</p>
+        <p>We've been notified and are working to fix the issue.</p>
+        <div class="action">
+            <button onclick="window.location.reload()">Try Again</button>
+        </div>
+        <div class="status-code">HTTP 500 - Internal Server Error</div>
+    </div>
+</body>
+</html>