Keeping kubernetes secrets in git with sealed secrets

When we start deploying in Kubernetes it is common to ask ourselves: where should we host the secrets, how do we deploy these secrets in a Continuous Delivery (CD) stream, is it possible to version the changes made in them, how do we deploy them in a CD stream, is it possible to version the changes made in them? We are going to clarify these questions and some others in this post.

At DataDope, to facilitate the management, deployment and versioning of the secrets, we use Sealed

Secrets. Sealed Secrets is a tool developed by Bitnami (VMWare).

Sealed Secrets uses asymmetric encryption (public key and private key). Once the Sealed Secrets Controller is deployed, it will generate a pair of keys that we will use to encrypt or decrypt our secrets.

• Public key: This key will be used to generate the Sealed Secrets from a secret. This must be downloaded by each of the deployment users in the Kubernetes cluster.
• Private key: It will be in charge of decrypting the secrets from the Sealed Secrets previously encrypted with the public key. This key will initially only be used by the Sealed Secret Controller. We must avoid downloading this key from the controller as it could compromise all the encrypted secrets.

The Sealed Secret can be versioned in Git. This procedure allows to centralise in a single repository all the manifests that make up the continuous delivery (CD) of our resources to Kubernetes, for example, using tools such as ArgoCD or FluxCD.

Installation

First, you need to deploy Sealed Secrets Controller. You can find all available versions at https://github.com/bitnami-labs/sealed-secrets/releases.

$ export SEALED_SECRETS_VERSION=0.18.0 $ curl -sSLo controller.yaml "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${SEALED_SECRETS_VERSION}/controller.yaml"

$ kubectl apply -f controller.yaml 
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
deployment.apps/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
service/sealed-secrets-controller created

The next step would be to install kubeseal. It is a command line tool that allows us to encrypt and decrypt Sealed Secrets.

$ curl -sSLo kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${SEALED_SECRETS_VERSION}/kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz"
$ tar xzf kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz
$ cp kubeseal /usr/local/bin/kubeseal

Use

We download the public key that we will use to encrypt the secrets.

$ kubeseal --fetch-cert --controller-name sealed-secrets-controller > sealed-secrets.crt

Create the secret to be encrypted.

$ cat secret.yaml
apiVersion: v1
data:
  password: YmFyCg==
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/managed: "true"
    sealedsecrets.bitnami.com/namespace-wide: "true"
  name: foo
  namespace: default
type: Opaque

• sealedsecrets.bitnami.com/managed: With this annotation we indicate to the controller that it can overwrite the secret in the specified namespace if it already exists with the same name.
• sealedsecrets.bitnami.com/namespace-wide: The secret can be freely renamed to the name space defined in the secret.

We encrypt the secret with kubeseal using the public key.

$ kubeseal --format yaml --cert sealed-secrets.crt <secret.yaml> sealed-secret.yaml

Example of an encrypted secret:

$ cat sealed-secret.yaml 
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  creationTimestamp: null
  name: foo
  namespace: default
spec:
  encryptedData:
    password: AgBT3O2pIbaeljVJIT4JK4Y+gzyo3dvJDdP2Cb75MirELjzk54Tj2Vq+rh33rdphtRFglJNmIgzXVGYctXAHaJes+lbCD+KDlpJJkn3N4EB97ypgD3e+14fet9nvOLnZxCgCMgDesp8NEX94ek/TgemmZIjpLQiCjKq41CosgQG+Bxk/lbODv1gj2fr0wvwvDrfZExEj4R4Rw5B1fUhzxgqDc1XAX/mfTOUAfyS2nC7iUyaFLh2JefYD42GtSnSF3PUadO5hY26hhkYy5JiG1M1EtX7b91HdkqBeU2jfdaC11gcHWOhoWyMWl0X6bz9JTFhirfOYW3T1uADv4ySk9No9vmmtk9CfPLe+/uWVzfv0KI93VmP9Azwi476OBIHXPZuB+nOZfA57om7mIQUEFBdaJl3a/hCdTm4Fr61Jl4GJ8CoS73nEzpef++fo0CETxxh7F+LGUH+re2roUlxBgL7lKvIocZjkbOilfEA51vIUwEishkpBM6OfbTZiyVroMi0XISAULzZgszJXeafIN/leLQk3gLHlzPrHLaXr5TAaXf0JMOpopY9ibeSuovGYnnFcaNdDoaIeS5AUPp5/kdmYk6qg3V6YXxd/c9NfAS76NKBbXCTFBcnArkz7/zpfWlFYjHG0B+xwortFWReHToMTZ4ZWC51aRaRa4J43NWlL3Oh9ImRYxpPrjB0ukWK/wghoTBTk
  template:
    data: null
    metadata:
      annotations:
        sealedsecrets.bitnami.com/managed: "true"
        sealedsecrets.bitnami.com/namespace-wide: "true"
      creationTimestamp: null
      name: foo
      namespace: default
    type: Opaque

We deploy the encrypted secret on Kubernetes.

$ kubectl apply -f sealed-secret.yaml 
sealedsecret.bitnami.com/foo created

We check that the new Sealed Secret has been created…

$ kubectl get sealedsecrets
NAME   AGE
foo    2m5s

And its corresponding secret:

$ kubectl describe secret foo
Name:         foo
Namespace:    default
Labels:       <none>
Annotations:  sealedsecrets.bitnami.com/managed: true
              sealedsecrets.bitnami.com/namespace-wide: true

Type:  Opaque

Data
====
password:  4 bytes

The secret is now available for use in our cluster. In addition, it is also possible to download the private key if we need to decrypt a file. WARNING! This task should only be carried out in situations of extreme necessity. An improper use of the private key could compromise all the secrets of the cluster.

kubectl get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >sealed-secrets.key

Decoding a secret.

$ kubeseal --recovery-unseal --recovery-private-key sealed-secrets.key -o yaml<sealed-secret.yaml 
apiVersion: v1
data:
  password: YmFyCg==
kind: Secret
metadata:
  annotations:
    sealedsecrets.bitnami.com/managed: "true"
    sealedsecrets.bitnami.com/namespace-wide: "true"
  creationTimestamp: null
  name: foo
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: foo
    uid: ""
type: Opaque

Source: https://github.com/bitnami-labs/sealed-secrets