o11y

A DevOps shop showcase of observability in action.

Navigate this repository with the blog article !

Contents

A Terraform project that deploys to a Minikube cluster the following tools:

• Grafana
• Alloy
• Loki
• Kube-Prometheus-Stack
• Tempo
• Minio
• Pyroscope

It also deploys the following example services:

These need to be updated to use the Kubernetes Gateway !

• The WallAPI, a REST API written in Golang instrumented via OpenTelemetrySDK and profiled by Pyroscope via /pprof
• The WallClient, a front-end client written in React, auto-instrumented via Faro -> Grafana Agent -> Tempo & Loki

The project is moving towards a micro-service architecture, powered by Kubernetes Gateway implemented by Istio. The Gateway binds on ports 80 and 443 and controls access to the following services:

• auth: An authentication service that provides HTTP handlers POST /register and POST /login which will provide the user with JWTs. The JWT private/public keys are generated by Terraform and added to the .runtime/ folder. The service also provides a GET /.well-known/jwks.json endpoint which provides the public key to validate tokens.

• url-shortner: A simple url shortner service. The gateway requires request to this service to have a Authorization header providing the JWT.

Minikube

To test the complete stack, create a Minikube cluster then apply with Terraform.

The API Gateway requires minikube tunnel to be running in order for the LoadBalancer service to bind to an IP address. If the tunnel is not up, the Chart will fail to install and the terraform process won't be able to finish.

$ minikube start
$ minikube tunnel &> /dev/null & # runs tunnel in the background
...
$ terraform apply
...

Once applied, you may add the Ingresses for the various services to your /etc/hosts file

$ kubectl get ingress -n monitoring
NAME                               CLASS    HOSTS                 ADDRESS        PORTS   AGE
grafana                            <none>   grafana.local         192.168.49.2   80      10m
...

Note: On Windows, you may need to use the minikube tunnel command and use 127.0.0.1 instead !

You may now access services via your browser.