What is this?

The gitops stack is the deployment infrastructure of the FAForever project. Currently it only contains a subset of service, but in the long term, all services shall be migrate from faf-stack to here.

Branching

• main branch = Production state
• develop branch = Testing state

Until things are settled the current workflow is:

• Test changes on develop
• Cherry pick working stuff on to main
• Rebase and force push develop afterwards

Maintenance / Release management

This repository and the cluster are setup to do the following:

• Keel is used to update mutable auto-update images (e.g. nodebb:4.4 updating to 4.4.x releases)
• Renovate is used to create PRs when image tags are outdated (for pinned versions as well as mutable versions)

Especially for 3rd party apps we try to use release-line tags. For our own apps we use pinned versions for apps that require combined deployments, for other apps we might use a release branch like main. In some cases only the test cluster points to rolling-release tags, while prod uses pinned release.

Attention: Renovate does not detect version changes if prod/dev use helm value based release tags.

Bootstrapping

First, we need to make the cluster gitops-ready by installing ArgoCD:

kubectl create namespace argocd
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd -n argocd

Then you can connect to argocd using the credentials generated:

PW=$(kubectl get secret -n argocd argocd-initial-admin-secret -o "jsonpath={.data.password}" | base64 -d)

echo "Access argocd via http://localhost:8080"
echo "User: admin, password: $PW"

kubectl port-forward -n argocd svc/argocd-server 8080:80

Now, we need to deploy the appsets via

kubectl apply -f app-set-test.yaml

Many apps will fail to start, because the lack the secrets that will be generated by infisical. But infiscal needs be setup too. We use the cloud edition, but there is also a self hosted one we do not cover here. For our stack you need to create a service token in the web ui and add this as a secret in all affected namespaces:

for namespace in "faf-apps faf-ops argocd"; do
kubectl create secret generic "infisical-service-token" \
    -n "$namespace" \
    --from-literal=infisicalToken=<your-token-here>
done

Local Development Using Tilt

This repository aims to provide a ready-to-go Tilt setup for ephemeral FAF infrastructure local development on kubernetes.

Prerequisites

• Tilt
• Kubernetes (Docker Desktop or Minikube are recommended for those new to kubernetes) Tilt cluster setup
• helm must be installed to generate the k8s from the helm charts in the gitops repo
• For Windows users:
  • A bash program. By default git bash is used with an assumed installation directory of C:/Program Files/Git

Startup Services

In the root directory of the repository run tilt up. This will start all the faf services in the correct order. The status of each service can be viewed in the tilt UI by visiting http://localhost:10350. This is the control server for tilt where you can restart services or disable them for substitution by services you would like to run from source code as you actively develop them.

Development

To develop against the FAF infrastructure you can insert the service in tilt that you are actively developing by adding it to the local_services list in your tilt_config.json file or as --local_services when you run tilt up. This will proxy the connections in the cluster to that service to your locally running instance. Some tweaks may need to be made to the default configuration parameters in the source code. The proper values can be found in the configMaps in each of the services kubernetes deploy yaml files. You may need to specify your host-ip in the config or arguments in order to get the proxied connections to work

Test Data

The default test data that is loaded can be found in faf-db. You can load the data at anytime by manually triggering the popualte-db resource under the Database label. This needs to be done the first time for data to exist.

Endpoints

Locally traefik uses the same ingress routes as we do in production just with a default base domain of localhost. So accessible domains will be api.localhost, hydra.localhost etc. By default these should resolve to 127.0.0.1. If they do not you can manually modify your hosts file to add the local dns records. For convenience the resource create-hosts-file-content will output the necessary lines for the hosts file when run after every service has started.