Loading...
Install HashiCorp Vault on Kubernetes using Helm — Part 1
Marco Franssen
11 min read•2124 words
In this post, I'll show you how to deploy HashiCorp Vault on Kubernetes using Helm. We'll start with a local deployment for development and experimentation. Later in the series, we'll deploy a highly available setup on AWS using HashiCorp Consul and automatic unsealing with an AWS KMS key. I assume you're familiar with Vault, Helm, Kubernetes, and Consul, so I won't go into much detail about the tools themselves.
In this first article, we'll focus on deploying Vault in our local environment. Let's briefly introduce the tools we'll use to set the scene.
HashiCorp Vault
Manage Secrets and Protect Sensitive Data
Secure, store, and tightly control access to tokens, passwords, certificates, and encryption keys to protect secrets and other sensitive data using a UI, CLI, or HTTP API.
Helm
The package manager for Kubernetes
Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste.
Kubernetes
Production-Grade Container Orchestration, automated container deployment, scaling, and management
Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.
It groups the containers that make up an application into logical units for easy management and discovery. Kubernetes builds on 15 years of experience running production workloads at Google, combined with ideas and practices from the community.
Prerequisites
To follow this post on your development machine, I recommend installing the following tools.
- Docker Desktop
- Kubernetes (also ships with Docker Desktop)
- kubectl (also ships with Docker Desktop)
- Helm
A quick way to install these tools on your Mac is to use Homebrew.
brew install --cask docker # docker as well installs kubectl
brew install helmOn Windows, you can use Chocolatey to install the tools.
choco install docker-desktop kubernetes-helmIf you'd like to learn more about these tools, see the references at the end of this article.
Ready to rumble
Now that our tooling is in place, we can enable Kubernetes in Docker Desktop.
With everything in place, we can deploy some workloads to Kubernetes. I want to reuse HashiCorp's existing Helm charts to automate most of the deployment. First, we'll add the Helm repository.
$ helm repo add hashicorp https://helm.releases.hashicorp.com
https://helm.releases.hashicorp.com
$ helm repo list
NAME URL
hashicorp https://helm.releases.hashicorp.com
…Now that we've added the Helm repository, we can search it for charts.
$ helm search repo vault
NAME CHART VERSION APP VERSION DESCRIPTION
hashicorp/vault 0.13.0 1.7.3 Official HashiCorp Vault ChartThis output shows chart version 0.13.0, which uses version 1.7.3 of Hashicorp Vault. Let's try installing it on our Kubernetes cluster. First, let's check that we're connected to the correct cluster by requesting cluster-info.
$ kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
KubeDNS is running at https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxyAs you can see, we're connected to our local Kubernetes cluster. Now let's install the Helm chart in a namespace called my-vault.
$ helm -n my-vault install --create-namespace -g hashicorp/vault
NAME: vault-1625395823
LAST DEPLOYED: Sun Jul 4 12:50:25 2021
NAMESPACE: my-vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://www.vaultproject.io/docs/
Your release is named vault-1625395823. To learn more about the release, try:
$ helm status vault-1625395823
$ helm get manifest vault-1625395823The -g parameter generates a random name for the release. Go ahead and try the commands shown in the output to get more details about your Helm release. If a command doesn't work, add the namespace as we did in the install command. The manifest command shows all the Kubernetes resources created by this release. Now let's look at the pods.
$ kubectl -n my-vault get pods
NAME READY STATUS RESTARTS AGE
vault-1625395823-0 0/1 Running 0 3m42s
vault-1625395823-agent-injector-5d94d68689-fs58c 1/1 Running 0 3m42sAs you can see, one of the pods isn't ready. Let's find out why by looking at its logs.
$ kubectl -n my-vault logs vault-1625395823-0
==> Vault server configuration:
Api Address: http://10.1.0.118:8200
Cgo: disabled
Cluster Address: https://vault-1625395823-0.vault-1625395823-internal:8201
Go Version: go1.15.13
Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: file
Version: Vault v1.7.3
Version Sha: 5d517c864c8f10385bf65627891bc7ef55f5e827
2021-07-04T10:50:43.289Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
==> Vault server started! Log data will stream in below:
2021-07-04T10:50:50.440Z [INFO] core: security barrier not initialized
2021-07-04T10:50:50.440Z [INFO] core: seal configuration missing, not initialized
2021-07-04T10:50:55.521Z [INFO] core: security barrier not initialized
2021-07-04T10:50:55.521Z [INFO] core: seal configuration missing, not initialized
2021-07-04T10:51:00.424Z [INFO] core: security barrier not initialized
…Connect to the Vault CLI within the container
Vault hasn't been initialized yet. If you've worked with HashiCorp Vault before, you'll know initialization is the first step. We can use our local Vault CLI by configuring the VAULT_ADDR environment variable, or we can use exec to run the CLI inside the container. Accessing Vault from outside the Kubernetes cluster requires a little more work to expose its API, so let's try connecting from within the container first.
$ kubectl --namespace my-vault exec -it vault-1625395823-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.7.3
Storage Type file
HA Enabled false
command terminated with exit code 2
$ kubectl --namespace my-vault exec -it vault-1625395823-0 -- vault operator init
Unseal Key 1: upSmKPHsRR8ihCvjZ4VKJwRcaK9EiqOKPNjXGyjuv4NI
Unseal Key 2: WyQTh2JCDA3UsZ4QIgFI/K8qxpA8Dels9PoG12uf6Usq
Unseal Key 3: x1a05Yn5xWANyRb6aryTb2HM4XUKwOVX2jhkzQk+XELj
Unseal Key 4: JHwLBRfsBfYwN9Ts69JR+HJ9NHMRqu9RDx2cgPenwaeH
Unseal Key 5: 5cVmkn+nAhlhKDeQ/IM/6AUvsBWXhF4F60Gmuy0RuzZU
Initial Root Token: s.mGbFL0B7VpmA8eaYaOWNiOES
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.The Vault status confirms that our Vault is uninitialized and sealed. I then generated the unseal keys. Make sure you keep them secure at all times. Never share your own keys as I've done here in this post.
⚠️ Write down your secrets for use later in this article.
The vault operator init command supports additional parameters to fine-tune the key shares or distribute them among Keybase users. That last option is powerful: rather than giving one person all the keys for "nuclear control", you can require several team members to unlock the store.
Checking the status of our pods again, we now see that the Vault pod is also in a ready state.
$ kubectl -n my-vault get pods
NAME READY STATUS RESTARTS AGE
vault-1625395823-0 1/1 Running 0 9m37s
vault-1625395823-agent-injector-5d94d68689-fs58c 1/1 Running 0 9m37sConnect locally with the Vault CLI
If you have the Vault CLI installed locally, you can connect by configuring the API endpoint. To install Vault, run brew install vault (macOS) or choco install vault (Windows). To find the API endpoint, we need to query the services in our Kubernetes deployment.
If you don't want to install the CLI locally, you can open a shell in the container (kubectl --namespace my-vault exec -it vault-1625395823-0 -- sh). For the following commands, skip setting the VAULT_ADDR environment variable and run them without it. Keep following along to access the Vault UI in your browser later in this article.
$ kubectl -n my-vault get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
vault-1625395823 ClusterIP 10.104.86.249 <none> 8200/TCP,8201/TCP 20h
vault-1625395823-agent-injector-svc ClusterIP 10.109.66.228 <none> 443/TCP 20h
vault-1625395823-internal ClusterIP None <none> 8200/TCP,8201/TCP 20hWe don't have an external IP yet. Let's update our Helm deployment to expose the API outside the Kubernetes cluster. The Vault Helm chart lists the options we can use to customize the deployment. We'll enable ingress to access HashiCorp Vault from outside the cluster.
$ echo "server:
ingress:
enabled: true
hosts:
- host: localhost
paths: []
" > customize-vault.yaml
$ helm -n my-vault upgrade vault-1625395823 hashicorp/vault -f customize-vault.yaml
W0705 10:06:57.709957 10328 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0705 10:06:57.880034 10328 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0705 10:06:57.884498 10328 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Release "vault-1625395823" has been upgraded. Happy Helming!
NAME: vault-1625395823
LAST DEPLOYED: Mon Jul 5 10:06:57 2021
NAMESPACE: my-vault
STATUS: deployed
REVISION: 2
NOTES:
Thank you for installing HashiCorp Vault!
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
https://www.vaultproject.io/docs/
Your release is named vault-1625395823. To learn more about the release, try:
$ helm status vault-1625395823
$ helm get manifest vault-1625395823
$ VAULT_ADDR=http://localhost/ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.7.3
Storage Type file
HA Enabled falseWe can now access Vault through the configured ingress, and its status confirms that it's initialized. Let's unseal it so we can start using it. For this unseal configuration, you'll need to provide three of the five unseal keys.
$ VAULT_ADDR=http://localhost/ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 0ff568f3-1327-6c99-581a-206271068364
Version 1.7.3
Storage Type file
HA Enabled false
$ VAULT_ADDR=http://localhost/ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 0ff568f3-1327-6c99-581a-206271068364
Version 1.7.3
Storage Type file
HA Enabled false
$ VAULT_ADDR=http://localhost/ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.7.3
Storage Type file
Cluster Name vault-cluster-a4e795d8
Cluster ID 361c5ea0-2ee4-43ed-132a-cd180de0aca9
HA Enabled falseNow that Vault is unsealed, we can use it through either the CLI or the Vault UI. Go ahead and try accessing it from your browser.
Sign in using the RootToken (shown earlier when we ran init) and experiment with configuring a secret store. You can also sign in on the command line and explore Vault through the CLI. The workflow below logs in and then stores a value in the cubbyhole store.
$ VAULT_ADDR=http://localhost/ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.yqqejg7VCk7J5sOsPly0WSN7
token_accessor c72GA4zI7Jh4JHKY9BXvf8AZ
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
$ VAULT_ADDR=http://localhost/ vault write cubbyhole/awesome stuff=cool
Success! Data written to: cubbyhole/awesome
$ VAULT_ADDR=http://localhost/ vault list cubbyhole
Keys
----
awesome
$ VAULT_ADDR=http://localhost/ vault read cubbyhole/awesome
Key Value
--- -----
stuff coolCheck out your new secret in the UI as well. In the next article, I'll show you how to configure Vault in high-availability mode using Consul on an EKS cluster in AWS, and how to use AWS KMS to unseal Vault automatically.
While waiting for the next post, you might want to experiment further with this setup. You can also enable development mode in the chart. This creates a deployment with Vault unsealed by default and a root token defined in values.yaml. It's convenient for exploring Vault, but I don't recommend it for any non-development environment.
References
Thanks for reading, and see you in Part 2.
