Hello world

A single container sample that covers:

• ACR Tasks for building & testing container images
• ACR Helm Chart Repos
• ACR Tasks for deploying Helm Charts to AKS

While the readme may see a bit long, we attempt to follow best practices, such as using Azure Key Vault to store secrets.

Assumptions

• A current version of the az cli
• An existing AKS cluster configured
• Helm Client is installed

Fork the Repos

To complete the sample, git webhook creation on a repo you own will be required.

• Fork the following repositories:
  • github.com/demo42/helloworld
  • github.com/demo42/helloworld-deploy

Configure your local environment

• Edit ./env.sh to represent your specific environment and resource names

• CD into helloworld and apply the environment variables with source

  cd ./helloworld
  source ./env.sh

• Login to the az cli

  az login

• Configure a default registry, to avoid having to specify the registry in each az acr command. This uses the env.sh values set above.

  az configure --defaults acr=$ACR_NAME

Create a GitHub personal access token

To trigger a build on a commit, ACR Tasks needs a personal access token (PAT) to access the git repository.

• Navigate to the PAT creation page on GitHub at https://github.com/settings/tokens/new

• Enter a short description for the token, for example, "ACR Build Task Demo"

Public Repos:

Public repos require the following permissions:

• Under repo, enable repo:status and public_repo

  

Private Repos:

• Private repos: add

  • repo, enable repo:status, repo:repo_deployment, repo:public_repo, repo:invite

  • admin:repo_hook, enable write:repo_hook, read:repo_hook

    

Select the Generate token button

• Copy the generated token and paste into Key Vault

  

  az keyvault secret set \
      --vault-name $AKV_NAME \
      --name $GIT_TOKEN_NAME \
      --value 74fef000b0000a00f000000

Credentials

To perform an AKS update using Helm, a service principal is required to pull images from the registry and execute helm update. To avoid losing the credentials, while storing them securely, we'll create a service principal, saving the secrets to Azure Key Vault

# Create a service principal (SP) with:
# - registry pull permissions
# - cluster deploy permissions

# Create a SP with registry pull permissions, saving the created password to a Key Vault secret.
az keyvault secret set \
  --vault-name $AKV_NAME \
  --name $ACR_NAME-deploy-pwd \
  --value $(az ad sp create-for-rbac \
            --name $ACR_NAME-deploy \
            --scopes \
              $(az acr show \
                --name $ACR_NAME \
                --query id \
                --output tsv) \
            --role reader \
            --query password \
            --output tsv)

# Store the service principal ID, (username) in Key Vault
az keyvault secret set \
    --vault-name $AKV_NAME \
    --name $ACR_NAME-deploy-usr \
    --value $(az ad sp show \
              --id http://$ACR_NAME-deploy \
              --query appId --output tsv)

# Assign permissions required for Helm Update
az role assignment create \
  --assignee $(az ad sp show \
              --id http://$ACR_NAME-deploy \
              --query appId \
              --output tsv) \
  --role owner \
  --scope $(az aks show \
              --resource-group $AKS_RESOURCE_GROUP \
              --name $AKS_CLUSTER_NAME \
              --query "id" \
              --output tsv)

# Save the tenant for az login --service-principal
az keyvault secret set \
    --vault-name $AKV_NAME \
    --name $ACR_NAME-tenant \
    --value $(az account show \
              --query tenantId \
              -o tsv)

Local (pre-commit) Build

One of the great things about ACR Tasks is the ability to run a quick task validating the work, before committing to source control.

With configurations complete, create a quick build* to validate the configurations

• Using ACR Tasks, execute a quick build

  az acr build -t helloworld:{{.Run.ID}} . 

• List images available, including the newly built image:

  az acr repository show-tags --repository helloworld

• List tags in lastupdate, descending order.

  az acr repository show-tags \
    --repository helloworld \
    --orderby time_desc \
    --detail \
    --query "[].{Tag:name,LastUpdate:lastUpdateTime}"

Deploy the Initial Hello World image to AKS

Before we automate helm chart updates, an initial seeding of the app is required as Helm uses separate commands for install and update.

• Initialize the helm client environment.

  helm init --client-only

• Add a helm repo, which refers to the Azure Container Registry, then list the local Helm repos.

  az acr helm repo add
  helm repo list

  output:

  NAME    URL
  stable  https://kubernetes-charts.storage.googleapis.com
  local   http://127.0.0.1:8879/charts
  demo42  https://demo42.azurecr.io/helm/v1/repo

  Note: By setting az configure --defaults acr=demo42, az acr helm repo add --name demo42 isn't required

• Package the helm chart from helloworld-deploy helloworld-deploy, which should be cloned alongside the helloworld folder

  helm package \
    --version 1.0.0 \
    ../helloworld-deploy/helm/helloworld

• Push the packaged helm chart to the registry

  az acr helm push ./helloworld-1.0.0.tgz
  

• Refresh the local Helm index

  az acr helm repo add

• Fetch the Helm Chart

  helm fetch --untar $ACR_NAME/helloworld --untardir ./charts

• Set the TAG, based on the most recent run-id, excluding latest

  az acr repository show-tags \
      --repository helloworld \
      --orderby time_desc \
      --detail \
      --query "[].{Tag:name,LastUpdate:lastUpdateTime}"
  
  TAG=___

• Helm install the initial deployment

  Note: using upgrade with the --install flag allows one command to be used for both install and upgrade. If the named deployment doesn't exist, it will be created. If it does exist, it will be updated.

  helm upgrade helloworld ./charts/helloworld \
  --install \
  --set helloworld.host=$HOST \
  --set helloworld.image=$ACR_NAME.azurecr.io/helloworld:$TAG \
  --set imageCredentials.registry=$ACR_NAME.azurecr.io \
  --set imageCredentials.username=$(az keyvault secret show \
                                         --vault-name $AKV_NAME \
                                         --name $ACR_NAME-pull-usr \
                                         --query value -o tsv) \
  --set imageCredentials.password=$(az keyvault secret show \
                                         --vault-name $AKV_NAME \
                                         --name $ACR_NAME-pull-pwd \
                                         --query value -o tsv)

• Query for an EXTERNAL_IP address to be provisioned.

  kubectl get svc

• Repeat until helloworld-nginx-ingress-controller returns a valid IP under EXTERNAL-IP

  NAME                                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
  helloworld-nginx-ingress-controller        LoadBalancer   10.0.235.75    40.117.59.143   80:30120/TCP,443:30141/TCP   21m
  helloworld-nginx-ingress-default-backend   ClusterIP      10.0.51.179    <none>          80/TCP                       21m
  helloworld-svc                             ClusterIP      10.0.62.141    <none>          80/TCP                       21m

Assign the static IP a dns name

To browse the website by a DNS name, find and assign through the Azure Portal.

• Navigate to https://portal.azure.com/

• Navigate to resource groups

• Filter resources groups to those starting with MC_

  

• Click into the resource group

• Click into the public ips, finding the one that has the same IP address as identified above

  

• Configure (1) the Public IP, assigning a notable dns name (2)

  

• Browse the site, using the DNS name

  

At this point, you should have a fully deployed HelloWorld app that looks similar to the above. Proceed to the next step to automate container builds

Automate Build and Deploy

With a basic/manual build and helm install complete, we'll transition to building & deploying an image to an AKS cluster.

ACR Tasks support multi-step operations, including the execution of a graph of containers.

The ./acr-task.yaml file represents the graph of steps executed:

• build the helloworld image, with two tags
• run the newly built image, in the task environment, detaching so a quick test can be performed
• run a quick functional test, using a curl image, passing it the url of the helloworld image. The url is based on the id: of the task.

Note: the curl doesn't actually validate the results, yet. This is a good TODO:

• push the validated image to the registry
• run the helm image, used for helm deployments.

Configuring AKS Permissions

To achieve a Helm Chart deployment, permissions to the AKS cluster are required. Since this is a headless service, the previously configured service principal is used.

• Run a quick-task over the local source, validating the build and helm deploy.

  az acr run \
    -f acr-task.yaml \
    . \
    --set CLUSTER_NAME=$AKS_CLUSTER_NAME \
    --set CLUSTER_RESOURCE_GROUP=$AKS_RESOURCE_GROUP \
    --set TENANT=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-tenant \
              --query value -o tsv) \
    --set SP=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-deploy-usr \
              --query value -o tsv) \
    --set PASSWORD=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-deploy-pwd \
              --query value -o tsv) \
    --registry $ACR_NAME 

Automatically build helloworld

With a quick build complete, configure an automated build that triggers on git commits and base image updates.

• Create an ACR Task with a set of variables used within the task, such as the AKS name and a service principal used for accessing AKS.

  az acr task create \
    -n helloworld \
    -f acr-task.yaml \
    --context $GIT_HELLOWORLD \
    --git-access-token $(az keyvault secret show \
                  --vault-name $AKV_NAME \
                  --name $GIT_TOKEN_NAME \
                  --query value -o tsv) \
    --set CLUSTER_NAME=$AKS_CLUSTER_NAME \
    --set CLUSTER_RESOURCE_GROUP=$AKS_RESOURCE_GROUP \
    --set-secret TENANT=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-tenant \
              --query value -o tsv) \
    --set-secret SP=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-deploy-usr \
              --query value -o tsv) \
    --set-secret PASSWORD=$(az keyvault secret show \
              --vault-name ${AKV_NAME} \
              --name $ACR_NAME-deploy-pwd \
              --query value -o tsv) \
    --registry $ACR_NAME

  With future Task enhancements, a Microsoft Identity (MSI) can be associated with a Task, avoiding the need configure service principal details above.

Commit a code change

• Monitor the current builds using a bash environment, including Azure cloud shell

  watch -n1 az acr task list-runs

• View the current executing task, without having to specify the --run-id

  az acr build-task logs

• View a specific task by the --run-id

  az acr build-task logs --run-id aabh

• View the update in the AKS Cluster

  TODO: navigate to the helloworld image, describing how to get the url

Base Image Updates

• Update the base image

  docker build -t baseimages/node:9 \
    -f node-jessie.Dockerfile \
    .

• Switch the dockerfile to -alpine

  Update the base image for Apline

  docker build -t jengademos.azurecr.io/baseimages/node:9-alpine -f node-alpine.Dockerfile .
  docker push jengademos.azurecr.io/baseimages/node:9-alpine

Update Demo42 Backcolor

docker tag jengademos.azurecr.io/baseimages/microsoft/aspnetcore-runtime:linux-2.1-azure \
  jengademos.azurecr.io/baseimages/microsoft/aspnetcore-runtime:linux-2.1
docker push \
  jengademos.azurecr.io/baseimages/microsoft/aspnetcore-runtime:linux-2.1

Deploy to AKS

• Get the cluster you're working with

  az aks list

• get credentials for the cluster

  az aks get-credentials -n [name] -g [group]

• Set variables

  export HOST=http://demo42-helloworld.eastus.cloudapp.azure.com/
  export ACR_NAME=jengademos
  export TAG=aa42
  export AKV_NAME=jengademoskv

• Deploy with Helm

  Set the

  helm install ./release/helm/ -n helloworld \
  --set helloworld.host=$HOST \
  --set helloworld.image=jengademos.azurecr.io/demo42/helloworld:$TAG \
  --set imageCredentials.registry=$ACR_NAME.azurecr.io \
  --set imageCredentials.username=$(az keyvault secret show \
                                         --vault-name $AKV_NAME \
                                         --name $ACR_NAME-pull-usr \
                                         --query value -o tsv) \
  --set imageCredentials.password=$(az keyvault secret show \
                                         --vault-name $AKV_NAME \
                                         --name $ACR_NAME-pull-pwd \
                                         --query value -o tsv)

## Helm Package, push
helm package \
    --version 1.0.1 \
    ./helm/helloworld

az acr helm push \
    ./helloworld-1.0.1.tgz \
    --force -o table

## Update the local cache
az acr helm repo add

helm fetch demo42/helloworld

helm repo list

## Upgrade
```sh
helm upgrade helloworld ./helm/helloworld/ \
  --reuse-values \
  --set helloworld.image=demo42.azurecr.io/helloworld:$TAG

Create the webhook header

Create a value in Key Vault to save for future reference

az keyvault secret set \
  --vault-name $AKV_NAME \
  --name demo42-helloworld-webhook-auth-header \
  --value "Authorization: Bearer "[value]

Create ACR Webhook for deployments

az acr webhook create \
  -r $ACR_NAME \
  --scope demo42/helloworld:* \
  --actions push \
  --name demo42HelloworldEastus \
  --headers Authorization=$(az keyvault secret show \
                            --vault-name $AKV_NAME \
                            --name demo42-helloworld-webhook-auth-header \
                            --query value -o tsv) \
  --uri http://jengajenkins.eastus.cloudapp.azure.com/jenkins/generic-webhook-trigger/invoke

Troubleshooting

• Verifying the registry credentials are being retrieved

  echo imageCredentials.username=$(az keyvault secret show \
                                          --vault-name $AKV_NAME \
                                          --name $ACR_NAME-pull-usr \
                                          --query value -o tsv) \
    imageCredentials.password=$(az keyvault secret show \
                                          --vault-name $AKV_NAME \
                                          --name $ACR_NAME-pull-pwd \
                                          --query value -o tsv)

• Resetting the registry credentials (secret)

  helm upgrade $APP_NAME ./charts/helloworld \
    --reuse-values \
    --set helloworld.image=$ACR_NAME.azurecr.io/helloworld:$TAG \
    --set imageCredentials.registry=$ACR_NAME.azurecr.io \
    --set imageCredentials.username=$(az keyvault secret show \
                                          --vault-name $AKV_NAME \
                                          --name $ACR_NAME-pull-usr \
                                          --query value -o tsv) \
    --set imageCredentials.password=$(az keyvault secret show \
                                          --vault-name $AKV_NAME \
                                          --name $ACR_NAME-pull-pwd \
                                          --query value -o tsv)

• Resetting the registry credentials, with the kubectl cli

  kubectl create secret \
    docker-registry acrdemossecret \
    --docker-server=acrdemos.azurecr.io \
    --docker-username=$(az keyvault secret show \
                          --vault-name $AKV_NAME \
                          --name $ACR_NAME-pull-usr \
                          --query value -o tsv) \
   --docker-password=$(az keyvault secret show \
                          --vault-name $AKV_NAME \
                          --name $ACR_NAME-pull-pwd \
                          --query value -o tsv) \
   --docker-email=dont@bother.me

• Verifying the registry credentials work

  docker login $ACR_NAME.azurecr.io \
    -u $(az keyvault secret show \
          --vault-name $AKV_NAME \
          --name $ACR_NAME-pull-usr \
          --query value -o tsv) \
    -p $(az keyvault secret show \
          --vault-name $AKV_NAME \
          --name $ACR_NAME-pull-pwd \
          --query value -o tsv)
  
  docker pull $ACR_NAME.azurecr.io/helloworld:$TAG

az acr run --cmd "orca run $Registry/base-artifacts/acr/helm --help" /dev/null
az acr run --cmd "orca run demo42t.azurecr.io/base-artifacts/bash:latest /bin/ls -l" /dev/null
az acr import \
      --name ${ACR_NAME} \
      --source demo42upstream.azurecr.io/mcr/acr/helm:v3 \
      -t base-artifacts/acr/helm:v3 \
      -t base-artifacts/acr/helm:latest \
      -t base-artifacts/acr/helm:v3-$ID \
      --force