When we start deploying in Kubernetes it is common to ask ourselves: where should we host the secrets, how do we deploy these secrets in a Continuous Delivery (CD) stream, is it possible to version the changes made in them, how do we deploy them in a CD stream, is it possible to version the changes made in them? We are going to clarify these questions and some others in this post.
At DataDope, to facilitate the management, deployment and versioning of the secrets, we use Sealed
Secrets. Sealed Secrets is a tool developed by Bitnami (VMWare).
Sealed Secrets uses asymmetric encryption (public key and private key). Once the Sealed Secrets Controller is deployed, it will generate a pair of keys that we will use to encrypt or decrypt our secrets.
- Public key: This key will be used to generate the Sealed Secrets from a secret. This must be downloaded by each of the deployment users in the Kubernetes cluster.
- Private key: It will be in charge of decrypting the secrets from the Sealed Secrets previously encrypted with the public key. This key will initially only be used by the Sealed Secret Controller. We must avoid downloading this key from the controller as it could compromise all the encrypted secrets.
The Sealed Secret can be versioned in Git. This procedure allows to centralise in a single repository all the manifests that make up the continuous delivery (CD) of our resources to Kubernetes, for example, using tools such as ArgoCD or FluxCD.
Installation
First, you need to deploy Sealed Secrets Controller. You can find all available versions at https://github.com/bitnami-labs/sealed-secrets/releases.
$ export SEALED_SECRETS_VERSION=0.18.0 $ curl -sSLo controller.yaml "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${SEALED_SECRETS_VERSION}/controller.yaml"
$ kubectl apply -f controller.yaml clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created clusterrole.rbac.authorization.k8s.io/secrets-unsealer created serviceaccount/sealed-secrets-controller created rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created role.rbac.authorization.k8s.io/sealed-secrets-key-admin created role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created deployment.apps/sealed-secrets-controller created customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created service/sealed-secrets-controller created
The next step would be to install kubeseal. It is a command line tool that allows us to encrypt and decrypt Sealed Secrets.
$ curl -sSLo kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${SEALED_SECRETS_VERSION}/kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz" $ tar xzf kubeseal-${SEALED_SECRETS_VERSION}-linux-amd64.tar.gz $ cp kubeseal /usr/local/bin/kubeseal
Use
We download the public key that we will use to encrypt the secrets.
$ kubeseal --fetch-cert --controller-name sealed-secrets-controller > sealed-secrets.crt
Create the secret to be encrypted.
$ cat secret.yaml apiVersion: v1 data: password: YmFyCg== kind: Secret metadata: annotations: sealedsecrets.bitnami.com/managed: "true" sealedsecrets.bitnami.com/namespace-wide: "true" name: foo namespace: default type: Opaque
- sealedsecrets.bitnami.com/managed: With this annotation we indicate to the controller that it can overwrite the secret in the specified namespace if it already exists with the same name.
- sealedsecrets.bitnami.com/namespace-wide: The secret can be freely renamed to the name space defined in the secret.
We encrypt the secret with kubeseal using the public key.
$ kubeseal --format yaml --cert sealed-secrets.crt <secret.yaml> sealed-secret.yaml
Example of an encrypted secret:
$ cat sealed-secret.yaml apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: annotations: sealedsecrets.bitnami.com/namespace-wide: "true" creationTimestamp: null name: foo namespace: default spec: encryptedData: password: AgBT3O2pIbaeljVJIT4JK4Y+gzyo3dvJDdP2Cb75MirELjzk54Tj2Vq+rh33rdphtRFglJNmIgzXVGYctXAHaJes+lbCD+KDlpJJkn3N4EB97ypgD3e+14fet9nvOLnZxCgCMgDesp8NEX94ek/TgemmZIjpLQiCjKq41CosgQG+Bxk/lbODv1gj2fr0wvwvDrfZExEj4R4Rw5B1fUhzxgqDc1XAX/mfTOUAfyS2nC7iUyaFLh2JefYD42GtSnSF3PUadO5hY26hhkYy5JiG1M1EtX7b91HdkqBeU2jfdaC11gcHWOhoWyMWl0X6bz9JTFhirfOYW3T1uADv4ySk9No9vmmtk9CfPLe+/uWVzfv0KI93VmP9Azwi476OBIHXPZuB+nOZfA57om7mIQUEFBdaJl3a/hCdTm4Fr61Jl4GJ8CoS73nEzpef++fo0CETxxh7F+LGUH+re2roUlxBgL7lKvIocZjkbOilfEA51vIUwEishkpBM6OfbTZiyVroMi0XISAULzZgszJXeafIN/leLQk3gLHlzPrHLaXr5TAaXf0JMOpopY9ibeSuovGYnnFcaNdDoaIeS5AUPp5/kdmYk6qg3V6YXxd/c9NfAS76NKBbXCTFBcnArkz7/zpfWlFYjHG0B+xwortFWReHToMTZ4ZWC51aRaRa4J43NWlL3Oh9ImRYxpPrjB0ukWK/wghoTBTk template: data: null metadata: annotations: sealedsecrets.bitnami.com/managed: "true" sealedsecrets.bitnami.com/namespace-wide: "true" creationTimestamp: null name: foo namespace: default type: Opaque
We deploy the encrypted secret on Kubernetes.
$ kubectl apply -f sealed-secret.yaml sealedsecret.bitnami.com/foo created
We check that the new Sealed Secret has been created…
$ kubectl get sealedsecrets NAME AGE foo 2m5s
And its corresponding secret:
$ kubectl describe secret foo Name: foo Namespace: default Labels: <none> Annotations: sealedsecrets.bitnami.com/managed: true sealedsecrets.bitnami.com/namespace-wide: true Type: Opaque Data ==== password: 4 bytes
The secret is now available for use in our cluster. In addition, it is also possible to download the private key if we need to decrypt a file. WARNING! This task should only be carried out in situations of extreme necessity. An improper use of the private key could compromise all the secrets of the cluster.
kubectl get secret -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >sealed-secrets.key
Decoding a secret.
$ kubeseal --recovery-unseal --recovery-private-key sealed-secrets.key -o yaml<sealed-secret.yaml apiVersion: v1 data: password: YmFyCg== kind: Secret metadata: annotations: sealedsecrets.bitnami.com/managed: "true" sealedsecrets.bitnami.com/namespace-wide: "true" creationTimestamp: null name: foo namespace: default ownerReferences: - apiVersion: bitnami.com/v1alpha1 controller: true kind: SealedSecret name: foo uid: "" type: Opaque