Install HashiCorp Vault on Kubernetes using Helm — Part 2

In Part 1, we set up the prerequisites and ran HashiCorp Vault on a local Kubernetes cluster. This time, we'll deploy Vault on an AWS EKS cluster in high-availability mode, using HashiCorp Consul for storage and AWS KMS to unseal Vault automatically.

First, let's look at the new tools we're introducing. If you haven't read Part 1, I recommend starting there for an introduction to the basics and the tools we'll build on here.

AWS

Amazon Web Services is an Amazon subsidiary that provides on-demand cloud computing platforms and APIs to individuals, companies, and governments on a metered, pay-as-you-go basis. We'll use two of its services to deploy our highly available Vault cluster on Kubernetes.

EKS

Amazon Elastic Kubernetes Service (Amazon EKS) gives you the flexibility to start, run, and scale Kubernetes applications in the AWS cloud or on-premises. Amazon EKS helps you provide highly available, secure clusters and automates key tasks such as patching, node provisioning, and updates.

KMS

AWS Key Management Service (KMS) makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications. AWS KMS is a secure and resilient service that uses hardware security modules that have been validated under FIPS 140-2, or are in the process of being validated, to protect your keys. AWS KMS is integrated with AWS CloudTrail to provide you with logs of all key usage to help meet your regulatory and compliance needs.

HashiCorp Consul

Service Mesh for any runtime or cloud

Consul automates networking for simple and secure application delivery.

Prerequisites

• Terraform (optional, if you'd like to automate AWS resource creation)
• kubectl (also ships with Docker Desktop)
• Helm

Set up AWS

As mentioned above, we'll use two AWS products: EKS and KMS. EKS lets us deploy a highly available Kubernetes cluster. KMS lets us configure HashiCorp Vault to use hardware security modules for unsealing.

We'll also need a policy and role that allow our Vault deployment to access the KMS key.

To set up an EKS cluster, you can use the Terraform module. This module provides a kubeconfig output for connecting kubectl and helm to the cluster. Alternatively, you can write the configuration to a file using the write_kubeconfig variable. Once it's written to a file, you can use it from your terminal.

$ KUBECONFIG=./my-eks-cluster/kubeconfig kubectl cluster-info
 
Kubernetes control plane is running at https://85B852F6159465DEAA55460FFD0288D1.yl4.eu-west-1.eks.amazonaws.com
CoreDNS is running at https://85B852F6159465DEAA55460FFD0288D1.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

In this article, I'll focus on accessing the KMS key from our EKS cluster. Provisioning the EKS cluster itself is outside the scope of this post.

The rest of this setup assumes an existing EKS cluster. We'll use [Terraform][terraform] to create the KMS key and define the policy that lets our Vault deployment access it. (You could also create the resources shown in the Terraform configuration manually in the AWS console.)

The following Terraform configuration creates these resources.

resource "aws_kms_key" "vault_ha_cluster" {
  description = "Key for unsealing my Vault HA setup using Consul"
  policy = templatefile("${path.module}/policies/kms.json", {
    vault_arn = aws_iam_role.vault.arn
  })
  tags = {
    environment = local.environment
  }
}
 
resource "aws_kms_alias" "vault_ha_cluster" {
  name          = "alias/vault-ha-cluster"
  target_key_id = aws_kms_key.vault_ha_cluster.key_id
}
 
resource "aws_iam_role" "vault" {
  name = "eks-${local.environment}-vault"
 
  assume_role_policy = templatefile("${path.module}/policies/assume-role-oidc.json", {
    openid_connect_provider_arn = module.cluster.oidc.provider_arn
  })
}

These resources create a KMS key to unlock Hashicorp Vault, plus an alias so we can refer to the key by a human-readable name. We also create an IAM role that's authorized to access the key. Later, our HashiCorp Vault deployment will assume this role. We define one policy to allow the role to be assumed and another to configure fine-grained permissions for the KMS key. You might have noticed that we reference two policy files in json format. Let's look at those as well.

{
  "Version" : "2012-10-17",
  "Id" : "key-cloud-healthcare-poc-eks-dev-cluster",
  "Statement" : [ {
    "Sid": "Enable IAM User Permissions",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::759729069002:root"
    },
    "Action": "kms:*",
    "Resource": "*"
  }, {
    "Sid" : "Allow access eks dev-cluster",
    "Effect" : "Allow",
    "Principal" : {
      "AWS" : "${eks_worker_nodes_role_arn}"
    },
    "Action": [
      "kms:Decrypt",
      "kms:Encrypt",
      "kms:DescribeKey"
    ],
    "Resource" : "*"
  } ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Federated": "${openid_connect_provider_arn}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity"
    }
  ]
}

Both files contain variables that we inject through the Terraform definitions, referencing other Terraform resources. One reference points to my cluster module, which may differ in your setup. My cluster module uses the Terraform EKS module to provision the EKS cluster and an OIDC provider. Make sure you configure the arn for your OIDC provider. If you're not provisioning it through Terraform, you'll need to set it up separately.

Install HashiCorp Consul

HashiCorp Consul is HashiCorp's service mesh. We'll make our Vault cluster highly available by storing its state in Consul's key-value storage. We'll use Helm charts again, as we did in Part 1, but this time we'll deploy to our EKS cluster. The HashiCorp Helm repository also contains a Consul chart. I'll install Consul with the defaults, but you can review its values.yaml file to see how to fine-tune the deployment for your needs.

$ helm -n my-consul install --create-namespace consul hashicorp/consul
W0725 20:08:30.419969    4478 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0725 20:08:30.792426    4478 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
NAME: consul
LAST DEPLOYED: Sun Jul 25 20:08:30 2021
NAMESPACE: my-consul
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Consul!
 
Now that you have deployed Consul, you should look over the docs on using
Consul with Kubernetes available here:
 
https://www.consul.io/docs/platform/k8s/index.html
 
 
Your release is named consul.
 
To learn more about the release, run:
 
  $ helm status consul
  $ helm get all consul

Install Vault in HA mode

To install Vault in high-availability mode, we'll create a values file that configures Consul storage and KMS unsealing. This overrides the chart defaults for our HA deployment.

Let's create vault-ha.yaml with the settings for our deployment. Replace the variables with your IAM role, KMS key alias, and AWS region.

server:
  ingress:
    enabled: true
    hosts:
      - host: dev-vault.my-domain.com
        paths: []
 
  ui:
    enabled: true
    serviceType: LoadBalancer
 
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: $VAULT_KMS_ROLE
 
  ha:
    enabled: true
 
    config: |
      ui = true
 
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
 
      storage "consul" {
        path = "vault"
        address = "HOST_IP:8500"
      }
 
      service_registration "kubernetes" {}
 
      seal "awskms" {
        kms_key_id = "$VAULT_KMS_KEY_ID"
      }
 
  extraEnvironmentVars:
    AWS_REGION: $AWS_REGION

With this configuration file in place, we can deploy Vault to our EKS cluster.

$ helm -n my-vault install --create-namespace -f ./vault-ha.yaml my-vault hashicorp/vault
NAME: my-vault
LAST DEPLOYED: Sun Jul 25 20:22:32 2021
NAMESPACE: my-vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!
 
Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:
 
https://www.vaultproject.io/docs/
 
 
Your release is named my-vault. To learn more about the release, try:
 
  $ helm status my-vault
  $ helm get manifest my-vault

The last step is to initialize Vault.

$ kubectl --namespace my-vault exec -it vault-1625395823-0 -- vault operator init
Recovery Key 1: calc++d/XLg0O4o6N0eXDGYXggan0CZlRdWQLp4TYzsh
Recovery Key 2: i4LPPGJ36W+ZeF2TbeT1KCoyH2fGjBTA/Sx2siDm+Vxe
Recovery Key 3: G3+KaTAOExG+NVuKgz/CvqVedu90yypX3GEmQy8F4WB7
Recovery Key 4: BHpifWhHEmoYBe9nD/fN7AYrYbAke+zrH0kszsT44uDp
Recovery Key 5: bKHrdx+cnUQv0ix9FkTGVQ+a7B4Je/wDj3Z1T8E76ztD
 
Initial Root Token: s.CX1hTxMFCPP7NhvOOdK0YKRS
 
Success! Vault is initialized
 
Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

As you can see, we don't get unseal keys this time because we're using an AWS KMS key to unlock Vault. However, if we lose access to the KMS key, we can still use three of these five recovery keys to regain access to Vault. Make sure you store this information safely, preferably distributing the recovery keys across several places so the keys for "nuclear control" aren't all in one place.

Key Takeaways

Using policies and IAM roles, we can define fine-grained access to AWS resources for our Kubernetes workloads. There are several ways to achieve this, but I found this approach the most convenient and easiest to configure.

• An AWS IAM role provides fine-grained permissions for specified AWS resources through a policy.
• A Kubernetes workload assumes the IAM role to use the permissions defined in its attached policy.

This keeps your Kubernetes workloads from doing more in your AWS environment than they're supposed to. Always make the permissions as fine-grained as possible to limit security risks.

Using yaml files, we can override a Helm chart's defaults to customize a deployment. I recommend committing these values files to your Git repository as well.

References

• AWS EKS
• AWS KMS
• Terraform Module EKS
• https://github.com/hashicorp/vault-helm
• https://github.com/hashicorp/consul-helm

I hope you enjoyed this second part and gained some insight into deploying Helm charts with custom configurations and specific access to cloud resources.