Gitops using Helmsman to apply our Helm Charts to k8s

In my last blog series I have shown an example of deploying Hashicorp Vault on Kubernetes using Helm Charts (see references). This time I want to show you how to more easily integrate this into your … wait for it … :smile:, DevSecGitOps flow. Especially Helm charts help a lot in connecting the software part with our infrastructure / deployment (DevOps). Besides that we can embed all kind of security practices in our Helm charts like for example RBAC, Network policies etc. In this blog I want to show how we can make GitOps a bit easier using Helmsman, by defining our environments in the form of a configuration file.

What is Helmsman

Helmsman is a Helm Charts (k8s applications) as Code tool which allows you to automate the deployment/management of your Helm charts from version controlled code. Using Helmsman we can define which Helm charts we want to deploy as a composed whole. We can do this by defining all of our needs in a configuration file, which can either be in yaml or toml format.

Folks familiar with Terraform might appreciate this tool as it enables a similar workflow like you would be used to when using Terraform.

Prerequisuites

To get started we will ofcourse need the helmsman cli. Next to that Helmsman also requires helm-diff, a plugin for Helm.

terminal

brew install -y helmsman
helm plugin install https://github.com/databus23/helm-diff

Now the only thing left is that we write some yaml or toml if you prefer. So let's start by defining some local Kubernetes environment as a Helmsman definition.

Helmsman configuration file

Please take some time to explore the contents of the yaml, before I will start explaining its contents.

local.yaml

metadata:
  org: "marcofranssen.nl/research"
  maintainer: "Marco Franssen"
  description: "My Awesome Helmsman demo"
 
settings:
  globalMaxHistory: 5
 
helmRepos:
  bitnami: "https://charts.bitnami.com/bitnami"
  traefik: "https://helm.traefik.io/traefik"
  hashicorp: "https://helm.releases.hashicorp.com"
 
namespaces:
  kube-system:
    protected: true
  default:
    protected: true
  gateway:
    protected: false
  identity:
    protected: false
  secrets:
    protected: false
 
apps:
  traefik:
    namespace: "gateway"
    enabled: true
    chart: "traefik/traefik"
    valuesFile: "local/traefik.yaml"
    version: "10.1.1"
    priority: -4
    wait: true
  keycloak:
    namespace: "identity"
    enabled: true
    chart: "bitnami/keycloak"
    valuesFile: "local/keycloak.yml"
    version: "4.1.1"
    priority: -1
    wait: true
  vault:
    namespace: "secrets"
    enabled: true
    chart: hashicorp/vault
    version: "0.14.0"
    valuesFile: local/vault.yaml
    wait: true

In the metadata section we can describe our environment.

Next we configured the globalMaxHistory, which defines how many previous versions of Helm releases are tracked for rollback purposes.

In the helmRepos section we define the Helm repositories for our environment. Helmsman will take care of executing the helm repo add commands for the repos that are not yet added.

In the namespaces section we can define the namespaces where we would like to deploy our Helm charts to. Please note this can also be existing namespaces for deployments you did earlier. By protecting a namespace you can prevent accidental changes to those environments. Doing an update/upgrade of a existing deployment with the same name in such namespace would require you to unprotect the namespace first. For that reason I always protect the kube-system namespace as there Kubernetes core critical components are usually deployed. I also like to protect the default namespace as that is usually a namespace where I deploy some stuff to play arround with. Helmsman will take care of creating the namespaces that didn't exist yet using kubectl.

In the apps section we can define all the Apps we would like to add to our environment. I like to use Traefik as my ingress controller. Furthermore any modern environment also needs some identity service like an OIDC compliant authentication and authorization component which I can use to integrate my applications. Last but not least I also want to have a way to securily manage my application secrets. As you can see I'm deploying a Helm release, named traefik, into the gateway namespace. I'm also deploying Keycloak into the identity namespace and Vault into the secrets namespace. Helmsman is also capable of doing all these deployments in parallel if we would specify wait: false. Using priorities we will be able to control the order of some deployments by assigning them a negative priority they will install before the others. This can be convenient if some of your other apps depend on these services. To configure the specific apps we provide the valuesFile. Helmsman will take these values when invoking the helm install cli.

Ofcourse we would also add the Helm charts here for the products we are building. E.g. various microservices. Simply add your own helm repository to the list or define a relative path to a local Helm chart in the chart settings. E.g. chart: ./charts/my-app.

Workspace layout

Now we know a bit more on how to define an environment with multiple Helm releases we should also have a look at how we can structure our workspace to manage different environments for our project. Maybe you have multiple development and staging environments. Or your team would like to have specific setups for local development.

Lets say we would like to manage following environments:

local (commonly shared environment setup for local development by teammembers on a local Kubernetes environemnt, e.g. docker-for-mac or kind or minikube)
eks-dev (environment on EKS where you will continuously test your latest greatest changes to the product you are working on)
eks-staging (environment on EKS where you will deploy specific releases to test them before production)
eks-prod (environment on EKS where you will deploy your production releases)

To structure our workspace we could therefore create a folder layout like this.

terminal

$ tree .
.
├── README.md
├── eks-dev
│   ├── autoscaler.yaml
│   ├── consul.yaml
│   ├── external-dns.yaml
│   ├── keycloak.yaml
│   ├── traefik.yaml
│   └── vault.yaml
├── eks-staging
│   ├── autoscaler.yaml
│   ├── consul.yaml
│   ├── external-dns.yaml
│   ├── keycloak.yaml
│   ├── traefik.yaml
│   └── vault.yaml
├── eks-prod
│   ├── autoscaler.yaml
│   ├── consul.yaml
│   ├── external-dns.yaml
│   ├── keycloak.yaml
│   ├── traefik.yaml
│   └── vault.yaml
├── eks-dev.yml
├── eks-staging.yml
├── eks-prod.yml
├── local
│   ├── keycloak.yml
│   ├── traefik.yaml
│   └── vault.yaml
└── local.yml
 
4 directories, 24 files

In this layout I consider my dev, staging, prod to be strictly segregated on different kubernetes clusters. In case you would like to approach it differently by installing on the same cluster but e.g. use namespaces to segregate the environments you probably want to organize your files slightly different. Make sure though to protect the namespaces for the other environments to prevent mistakes.

You can do that by mentioning in your eks-dev environment the production and staging namespaces as protected and vice-versa in your eks-staging and eks-prod. This way you can not accidentally overwrite apps in those namespaces.

e.g.:

eks-dev.yaml

namespaces:
  kube-system:
    protected: true
  secrets-dev:
    protected: false
  secrets-staging:
    protected: true
  secrets-prod:
    protected: true

This would allow us to install Hashicorp Vault in different namespaces for the different environments.

Coming back to my setup where the environments are in completely isolated clusters we will not have to define the namespaces for other environments. To have a quick start you could for example copy paste your local.yaml and adjust the paths to the value files as well add some more deployments like I did to manage the DNS records, loadbalancing, autoscaling etc. I won't go into detail on how to configure these Charts specifically as that is beyond the scope of this article, however by exploring the charts documentation at Artifact Hub you should be able to manage that on your own.

Next to that I have also written a blog series on the basics of using Helm charts (part-1, part-2). As you might have noticed in the eks environment above I was also installing Hashicorp Consul. The reason for that is I'm using a High available Setup of Vault on EKS that requires Hashicorp Consul. On local I deploy Hashicorp Vault using a more minimal setup. In part-1 I cover the local setup of Hashicorp Vault and in part-2 I cover the High Available setup on EKS. Consider reading those articles to get some more insight on the basics of deploying a Helm chart.

Deploying a Helmsman environment

Now we know how to define a Helmsman configuration and how to organize our workspace and values files in an efficient manageble way we can easily deploy multiple Helm charts in one go using Helmsman.

terminal

$ helmsman -f local.yaml -apply
 
 _          _
| |        | |
| |__   ___| |_ __ ___  ___ _ __ ___   __ _ _ __
| '_ \ / _ \ | '_ ` _ \/ __| '_ ` _ \ / _` | '_ \
| | | |  __/ | | | | | \__ \ | | | | | (_| | | | |
|_| |_|\___|_|_| |_| |_|___/_| |_| |_|\__,_|_| |_| version: v3.7.2
A Helm-Charts-as-Code tool.
 
2021-08-08 11:45:56 INFO: validating environment variables in /Users/marco/code/my-infrastructure/local/keycloak.yml
2021-08-08 11:45:56 INFO: validating environment variables in /Users/marco/code/my-infrastructure/local/spire.yaml
2021-08-08 11:45:56 INFO: Parsed [[ ./local.yml ]] successfully and found [ 3 ] apps
2021-08-08 11:45:56 INFO: Validating desired state definition
2021-08-08 11:45:56 INFO: Setting up kubectl
2021-08-08 11:45:56 INFO: Setting up helm
2021-08-08 11:46:00 INFO: Setting up namespaces
2021-08-08 11:46:00 INFO: Getting chart information
2021-08-08 11:46:02 INFO: Charts validated.
2021-08-08 11:46:02 INFO: Preparing plan
2021-08-08 11:46:02 INFO: Acquiring current Helm state from cluster
2021-08-08 11:46:11 INFO: Checking if any Helmsman managed releases are no longer tracked by your desired state ...
2021-08-08 11:46:11 INFO: No untracked releases found
2021-08-08 11:46:11 NOTICE: -------- PLAN starts here --------------
2021-08-08 11:46:11 NOTICE: Release [ traefik ] version [ 10.1.1 ] will be installed in [ gateway ] namespace -- priority: -4
2021-08-08 11:46:11 NOTICE: Release [ keycloak ] version [ 4.1.1 ] will be installed in [ identity ] namespace -- priority: -1
2021-08-08 11:46:11 INFO: Release [ vault ] installed and up-to-date -- priority: 0
2021-08-08 11:46:11 NOTICE: -------- PLAN ends here --------------
2021-08-08 11:46:11 INFO: Executing plan
2021-08-08 11:46:11 NOTICE: Install release [ traefik ] version [ 10.1.1 ] in namespace [ traefik ]
…………………………………………

Ofcourse there are more flags we could use. For example to define the kubeconfig file to be used. This is conventient for connecting to a specific EKS cluster. When your environment is growing you not necessarly want to apply all the apps. In that case you can use the -target flag to define one or many specific apps. Simply type helmsman --help to learn about all the commandline options you can use.

For example to only destroy the keycloak and vault app in your eks-dev environment you can use the following command line.

terminal

helmsman -f eks-dev.yaml -kubeconfig ./my-eks-kubeconfig -destroy -target keycloak -target vault

Skipping the -apply or -destroy flags would print you the plan about to be executed. This is very usefull if you only want to check the changes upfront without applying them.

Now the only thing remaining is to commit all you configurations in Git and maybe build some CI/CD jobs arround these to continuously deploy your changes to your eks-dev environment. Once done, you can start by building your own helm charts for your own apps.

Bonus

To create helm charts you could simply start with the following commandline to start on a new Chart generated from some scaffold.

terminal

helm create charts/service-a
helm create charts/service-b -p my-scaffold

service-a in this case uses the default scaffold template, where for service-b I create a new Helm chart using my custom scaffold. Our local helm charts will now be in the charts folder. In Helmsman configurations you can now easily refer to these local helm charts to test them in your environment.

apps:
  my-service-a:
    namespace: "domain-a"
    enabled: true
    chart: ./charts/service-a
    version: "0.1.0"
    valuesFile: local/service-a.yaml
    wait: true
  the-service-b:
    namespace: "domain-b"
    enabled: true
    chart: ./charts/service-b
    version: "0.1.3"
    valuesFile: local/service-b.yaml
    wait: true

Once you published your charts to a Helm repository you could ofcourse also install them via that repository, so you don't depend on relative filepaths to your charts anymore.

helmRepos:
  marco: "https://marcofranssen.github.io/helm-charts/"
 
apps:
  my-service-a:
    namespace: "domain-a"
    enabled: true
    chart: marco/service-a
    version: "0.1.0"
    valuesFile: local/service-a.yaml
    wait: true
  the-service-b:
    namespace: "domain-b"
    enabled: true
    chart: marco/service-b
    version: "0.1.3"
    valuesFile: local/service-b.yaml
    wait: true