Going secretless and keyless with Spiffe Vault

Securing the software supply chain has been a hot topic these days. Many projects have emerged with the focus on bringing additional security to the software supply chain as well adding zero-trust capabilities to the infrastructure you are running the software on.

In this blogpost I want to introduce you to a small commandline utility (spiffe-vault) that enables a whole bunch of usecases like:

• Secretless deployments
• Keyless codesigning
• Keyless encryption

Spiffe-vault utilizes two projects to achieve the secretless and keyless approach of implementing these usecases. Respectively Spiffe and Hashicorp Vault.

The project was founded 25th of August 2021. Why am I blogging only now about spiffe-vault?

To be honest there is actually no good reason, except the fact I just didn't free up time to do so. Another thing is because I have been explicitly been asked by one of my followers (Batuhan Apaydın a.k.a. developer-guy) in the community to write about it.

Now at first this might sound a bit fuzzy, but hold on, I will first introduce you to the concepts and why I build this tool and then dive into the implementation details.

Lets start by looking at some threaths in the Software supply chain.

In this diagram created by the SLSA project we can see there are various places in our software supply chain where a potential Attack can take place. In many of these steps we usually involve secrets while implementing measures to protect ourself from supply chain attacks.

Examples of these secrets are:

• The credentials to authenticate to a container registry
• The passphrase for our signing/encryption key
• The API keys to run a integration test against an API
• The credentials to push our release asset to a artifacts storage
• The secrets to deploy our release to our dev/qa/prod environments
• …

So why do we need all these secrets? For many of these attacks we want to protect ourself by being able to verify all the things we do (Zero Trust). In Zero Trust we trust nothing, but we verify everything. In the remainder of this article I want to zoom in on one particular usecase of keyless codesigning to explain the use of spiffe-vault.

In general our signature on our container image or any other release asset is only as strong as the way we are able to protect our signing key. E.g. if this key would be compromised an attacker could use it to inject malicious bits in our software release, sign it and pretend like this release was done by the original maintainer of the project itself. For us that would mean we have to store this signing key securely. Basically we do not want to have other beside ourself get their hands on our signing keys. So for the sake someone does get their hands on it, we also always add a strong password on this signing key. Great so now we are good right!

Unfortunately no, what if I accidentally leak my password for this key as well? Yeah I hear you say, I should have stored my password securely somewhere in a credentials manager like Keepass, Lastpass or one of those many tools. But then to access this tool I would again require a password. So to protect this password I would have to store this somewhere else securely. I could go on and on infinitely with this story. We call this the bottom turtle problem. This is where the Spiffe project comes into the picture, to solve this Bottom turtle problem.

So what if I told you we could actually implement the usecase of codesigning without the need for a signing key neither the need for a phassphrase to access this signing key. This is what I refer to as keyless and secretless.

SPIFFE

With the SPIFFE project we can enable our infrastucture to give identities to our workloads that can be cryptographically verified to ensure that this workload is trusted and not any malicious activity. E.g. if we would be running a selfhosted GitHub actions runner we could register this GitHub runner to retrieve a SPIFFE ID. Using this SPIFFE ID we can now implement a secretless OIDC login flow with Hashicorp Vault that is cryptograhically verified before we can make use of any Hashicorp Vault features. In order to do so you would have to deploy Spire in your infrastructure as well which is the reference implementation of SPIFFE. It ships with a bunch of components that allows to hand out the SPIFFE IDs as well perform the cryptographic verification etc.

To deploy Spire on a Kubernetes infrastructure I also opensourced a Helm chart to ease the deployment off Spire on a Kubernetes cluster.

Hashicorp Vault

Another prerequisite is to have a Hashicorp Vault server deployed somewhere in your infrastructure. See this small piece of Terraform to configure the transit engine for the usecase of Codesigning and enabling the JWT authentication to allow our SPIFFE ID based authentication with Hashicorp Vault. For the not so Terraform known readers, it basically does the following.

vault secrets enable transit
vault write -f transit/keys/cosign type=ecdsa-p256
vault auth enable jwt
vault write auth/jwt/config oidc_discovery_url=http://spire-oidc.my-spire default_role=""
vault policy write local-policy -<<EOF
path "transit/keys" {
   capabilities = ["list"]
}
path "transit/keys/cosign" {
   capabilities = ["read"]
}
path "transit/hmac/cosign/*" {
   capabilities = ["update"]
}
path "transit/sign/cosign/*" {
   capabilities = ["update"]
}
path "transit/sign/cosign/sha1" {
   capabilities = ["deny"]
}
path "transit/verify/cosign" {
   capabilities = ["write"]
}
path "transit/verify/cosign/sha1" {
   capabilities = ["deny"]
}
EOF
vault write auth/jwt/role/local role_type=jwt user_claim=sub bound_audiences=CI bound_subject=spiffe://dev.localhost/ns/my-app/sa/my-app-spiffe-vault token_ttl=60s token_policies=local-policy

This enables the jwt auth method and configures to use the spire oidc endpoint that was deployed by the Spire Helm chart. Furthermore it configures a jwt role that gives access to use the transit engine. Using the policy we allow creating signatures and verifying them, but we deny the usage of the insecure sha1 hashing algorithm.

Now we can use this to use the cosign signing key in Hashicorp Vault to sign our docker images without the need to specify any secret in our CI pipeline.

$ export VAULT_ADDR=http://localhost:8200
$ eval "$(spiffe-vault auth -role local)"
$ cosign sign -key hashivault://cosign marcofranssen/busybox:latest
Pushing signature to: index.docker.io/marcofranssen/busybox:sha256-febcf61cd6e1ac9628f6ac14fa40836d16f3c6ddef3b303ff0321606e55ddd0b.sig
$ cosign verify -key hashivault://cosign marcofranssen/busybox:latest
 
Verification for marcofranssen/busybox:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.
 
[{"critical":{"identity":{"docker-reference":"index.docker.io/marcofranssen/busybox"},"image":{"docker-manifest-digest":"sha256:febcf61cd6e1ac9628f6ac14fa40836d16f3c6ddef3b303ff0321606e55ddd0b"},"type":"cosign container image signature"},"optional":null}]

As you saw in the Hashicorp Vault setup we choose to give the token only a lifetime of 60 seconds. Ofcourse for this particular usecase we could reduce that to just a couple of seconds, which reduces the amount of time someone could abuse it.

The spiffe-vault auth -role local part will only succeed if the given workload, server or docker container, has a SPIFFE ID. In the kubernetes setup that is handled by the k8s-registrar component.

Run it yourself

The entire example with all automation scripts and steps in detail can be viewed here. It allows you to deploy the entire setup on a Kubernetes cluster.

github.com/philips-labs/spiffe-vault/tree/main/example#readme

But what if you would like to run your own ec2 instance for a GitHub selfhosted runner that registers itself with the spire server? Best place to get you started is following this Guide. spiffe.io/docs/latest/try/getting-started-linux-macos-x

Wrapping up

So in summary spiffe-vault enables you to authenticate Hashicorp Vault on the commandline without the need for a password or any other form of secret. Once authenticated you can do any vault commands you usually do or simply consume the REST api to build any usecase on top of it. This is great for any CI or CD, however we could use this to retrieve secrets deploytime to inject them into for example your backend deployments I strongly recommend to make your backend simply SPIFFE aware and implement the few lines of code from this tool into your own Backend. So the backend can on the fly retrieve the secrets and encryption keys and mTLS certificates from Hashicorp Vault.

authPath := "jwt"
audience := "CI"
role := "dev"
 
jwt, err := spiffe.FetchJWT(ctx, audience)
if err != nil {
  return err
}
 
c, err := vault.NewClient(authPath)
if err != nil {
  return err
}
 
err = c.Authenticate(jwt, role)
if err != nil {
  return err
}
 
// Now utilize any hashicorp Vault features you need, e.g. encrypt some data or retrieve a secret

That way there will be no plain text secrets floating arround anywhere, even not in environment variables.

References

• spiffe-vault
• The complete example on how to use spiffe-vault
• Spire Helm chart

Background materials:

• Solving the bottom turtle
• SPIFFE
• Spire
• Kubernetes
• Hashicorp Vault