Local Kubernetes Stack with Loki Logging

A complete local development environment featuring:

• Kind - Local Kubernetes cluster
• Sample Go API - REST API with structured logging
• Loki - Log aggregation system
• Promtail - Log collector
• Grafana - Log visualization and dashboarding

Prerequisites

The following tools need to be installed on your system:

• Docker Desktop - Container runtime (required for Kind)

  • Download from: https://www.docker.com/products/docker-desktop
  • Make sure Docker is running before proceeding

• Homebrew - Package manager for macOS (if on macOS)

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Installed Tools

The following tools have been installed:

• ✅ kind (v0.31.0) - Kubernetes in Docker
• ✅ kubectl (v1.35.0) - Kubernetes CLI
• ✅ helm (v4.1.0) - Kubernetes package manager
• ✅ go - Go programming language

Project Structure

local_stack_merln/
├── app/                          # Go application
│   ├── main.go                   # REST API with logging
│   ├── go.mod                    # Go module definition
│   ├── go.sum                    # Go dependencies
│   └── Dockerfile                # Multi-stage Docker build
├── k8s/                          # Kubernetes manifests
│   ├── app/                      # Application manifests
│   │   ├── deployment.yaml       # App deployment
│   │   └── service.yaml          # App service (NodePort)
│   └── loki-stack/               # Loki stack configuration
│       └── values.yaml           # Helm values for Loki/Promtail/Grafana
├── kind/                         # Kind cluster configuration
│   └── kind-config.yaml          # Cluster config with port mappings
├── scripts/                      # Automation scripts
│   ├── setup.sh                  # Create Kind cluster
│   ├── deploy-loki.sh            # Deploy Loki stack
│   └── deploy-app.sh             # Build and deploy sample app
└── README.md                     # This file

Quick Start

1. Start Docker Desktop

Ensure Docker Desktop is running:

docker ps

2. Create the Kind Cluster

./scripts/setup.sh

This will:

• Create a Kind cluster named local-stack
• Configure port mappings for Grafana (30000) and the API (30080)
• Wait for the cluster to be ready

3. Deploy the Loki Stack

./scripts/deploy-loki.sh

This will:

• Add the Grafana Helm repository
• Install Loki, Promtail, and Grafana
• Configure Loki as a datasource in Grafana
• Wait for all pods to be ready

Access Grafana:

• URL: http://localhost:30000
• Username: admin
• Password: admin

4. Deploy the Sample Application

./scripts/deploy-app.sh

This will:

• Build the Docker image for the Go application
• Load the image into the Kind cluster
• Deploy the application to Kubernetes
• Create a NodePort service for external access

Access the API:

• Base URL: http://localhost:30080

API Endpoints

The sample Go application exposes the following endpoints:

Endpoint	Method	Description
/	GET	Home page with welcome message
/health	GET	Health check endpoint
/api/users	GET	Returns a list of sample users
/api/data	GET	Returns random system metrics
/api/random	GET	Random response with different log levels

Example Requests

# Home page
curl http://localhost:30080/

# Health check
curl http://localhost:30080/health

# Get users
curl http://localhost:30080/api/users

# Get sample data
curl http://localhost:30080/api/data

# Random endpoint (generates different log levels)
curl http://localhost:30080/api/random

Viewing Logs

Using kubectl

# View logs from all sample-api pods
kubectl logs -l app=sample-api -n default

# Follow logs in real-time
kubectl logs -f deployment/sample-api -n default

# View logs from a specific pod
kubectl logs <pod-name> -n default

Using Grafana

1. Open Grafana at http://localhost:30000
2. Login with admin / admin
3. Go to Explore (compass icon in the left sidebar)
4. Select Loki datasource
5. Use LogQL queries to filter logs:

# All logs from sample-api
{app="sample-api"}

# Error logs only
{app="sample-api"} |~ "ERROR"

# Warning logs only
{app="sample-api"} |~ "WARN"

# Logs from a specific endpoint
{app="sample-api"} |~ "/api/random"

Log Levels in the Application

The sample application generates logs at different levels:

• INFO - Normal operations, requests, responses
• WARN - Warning conditions
• ERROR - Error conditions (simulated in /api/random)

The /api/random endpoint is particularly useful for testing as it randomly generates different log levels.

Cluster Management

Check cluster status

# Get cluster info
kubectl cluster-info

# Get all nodes
kubectl get nodes

# Get all pods
kubectl get pods -A

# Get services
kubectl get svc -A

Access Kubernetes Dashboard (Optional)

# Deploy the dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml

# Create a proxy
kubectl proxy

# Access at:
# http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Troubleshooting

Docker not running

If you see errors about Docker:

# Check Docker status
docker ps

# If not running, start Docker Desktop
open -a Docker

Kind cluster creation fails

# Delete existing cluster
kind delete cluster --name local-stack

# Recreate
./scripts/setup.sh

Pods not starting

# Check pod status
kubectl get pods -A

# Describe a failing pod
kubectl describe pod <pod-name> -n default

# Check events
kubectl get events -n default --sort-by='.lastTimestamp'

Cannot access Grafana or API

# Verify port mappings
docker ps

# Check if services are running
kubectl get svc -A

# Verify NodePort services
kubectl get svc sample-api -n default
kubectl get svc loki-stack-grafana -n default

Logs not appearing in Grafana

# Check Promtail pods
kubectl get pods -l app.kubernetes.io/name=promtail -n default

# Check Promtail logs
kubectl logs -l app.kubernetes.io/name=promtail -n default

# Check Loki logs
kubectl logs -l app.kubernetes.io/name=loki -n default

# Verify Loki is receiving logs
kubectl port-forward svc/loki-stack 3100:3100 -n default
curl http://localhost:3100/ready

Clean Up

Delete the application

kubectl delete -f k8s/app/

Uninstall Loki stack

helm uninstall loki-stack -n default

Delete the Kind cluster

kind delete cluster --name local-stack

Remove Docker image

docker rmi sample-api:latest

Customization

Modify the Go Application

1. Edit app/main.go
2. Rebuild and redeploy:

   ./scripts/deploy-app.sh

Adjust Loki Configuration

1. Edit k8s/loki-stack/values.yaml
2. Update the Helm release:

   helm upgrade loki-stack grafana/loki-stack \
     -n default \
     -f k8s/loki-stack/values.yaml

Change Application Replicas

Edit k8s/app/deployment.yaml and modify the replicas field, then:

kubectl apply -f k8s/app/deployment.yaml

Additional Resources

• Kind Documentation
• Kubernetes Documentation
• Loki Documentation
• Grafana Documentation
• Go Documentation

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│                     Local Machine                        │
│                                                           │
│  ┌────────────────────────────────────────────────────┐ │
│  │              Kind Cluster (local-stack)             │ │
│  │                                                      │ │
│  │  ┌──────────────┐    ┌─────────────┐               │ │
│  │  │  Sample API  │    │   Promtail  │               │ │
│  │  │  (2 replicas)│───▶│ (Log Agent) │               │ │
│  │  │  Port: 30080 │    └──────┬──────┘               │ │
│  │  └──────────────┘           │                       │ │
│  │                              │ Push Logs            │ │
│  │                              ▼                       │ │
│  │                      ┌──────────────┐               │ │
│  │                      │     Loki     │               │ │
│  │                      │ (Log Storage)│               │ │
│  │                      └──────┬───────┘               │ │
│  │                             │                       │ │
│  │                             │ Query Logs           │ │
│  │                             ▼                       │ │
│  │                      ┌──────────────┐               │ │
│  │                      │   Grafana    │               │ │
│  │                      │ Port: 30000  │               │ │
│  │                      └──────────────┘               │ │
│  └──────────────────────────────────────────────────────┘ │
│                          ▲                                │
│                          │                                │
│            Access via localhost:30000/30080              │
└─────────────────────────────────────────────────────────┘

License

This is a sample project for local development and testing purposes.