Encrypting Kubernetes Secrets at Rest: A Step-by-Step Guide

Kubernetes provides a robust and secure platform for orchestrating containerized applications, but managing sensitive information like passwords, API keys, and certificates (stored as Kubernetes secrets) is critical. By default, secrets in Kubernetes are stored in plain text within etcd, the key-value store for the cluster. This makes it essential to encrypt secrets at rest to prevent unauthorized access to sensitive data.

In this comprehensive guide, we’ll explore how to encrypt Kubernetes secrets at rest using an EncryptionConfiguration. We will cover everything from setting up the encryption configuration file, applying changes, and verifying that secrets are now encrypted in etcd.

Why Encrypt Secrets at Rest?

When Kubernetes secrets are stored in etcd, they are not encrypted by default. This can be a security risk, as anyone with access to the etcd datastore could potentially read sensitive data. Encrypting secrets at rest ensures that even if an attacker gains access to the storage layer, they will not be able to read the encrypted data without the appropriate decryption keys.

Step 1: Accessing the Master Node

To apply encryption settings to your Kubernetes cluster, you first need to access the master node. If you are using Minikube (a popular tool for running Kubernetes locally), you can SSH into the Minikube VM:

minikube ssh

If you are using a different Kubernetes setup, use SSH to access the master node where etcd is running.

Step 2: Viewing the Contents of etcd

To understand how secrets are stored by default, you can view the contents of the etcd datastore. The data is stored in binary format, and you need to navigate to the appropriate location where the etcd data is stored:

• For a standard Kubernetes setup:

    sudo vi /var/lib/etcd/member/snap/db
  

• For a Minikube setup:

    sudo vi /var/lib/minikube/etcd/member/snap/db
  

Here, we use vi (or another text editor) to open the binary data file, and since the data is stored in binary format, you'll see non-human-readable characters.

Step 3: Searching for Secrets in etcd

To verify that secrets are stored in plain text, search for the key or value of a secret. In vi, type:

/keyorval

This search will reveal that secrets are indeed stored in plain text. To protect these secrets, we need to configure encryption at rest.

Step 4: Creating the Encryption Configuration File

To enable encryption, create an encryption configuration file on the master node. Let's create it at the following location:

sudo vi /etc/kubernetes/etcd-enc/etcd-encryption.yaml

Add the following content to the file:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: YWJjZGVmZ2hpamtsbW5vcA==
      - identity: {}

Explanation of the Encryption Configuration File

• apiVersion: apiserver.config.k8s.io/v1: This specifies the API version for the configuration.

• kind: EncryptionConfiguration: This defines the type of configuration as an encryption configuration.

• resources:: This section specifies which Kubernetes resources should be encrypted. In our case, we are encrypting only secrets.

• providers:: This section defines the encryption providers and their order of preference.

  • aescbc:: This specifies the use of the AES-CBC encryption method.

    • keys:: The encryption keys used by the aescbc provider.

      • name: key1: The name of the encryption key.

      • secret:: The base64-encoded secret key. In this case, YWJjZGVmZ2hpamtsbW5vcA== is a base64-encoded string.

  • identity:: This is a provider that stores data in plaintext. It acts as a fallback option if the aescbc provider is unavailable.

Key Requirements for the Encryption Key

The encryption key (secret) must be base64-encoded and of a certain length (16, 24, or 32 bytes for AES). If the key is not properly encoded, Kubernetes will throw an error. You can generate a base64-encoded secret using the following command:

echo -n 'mysecretkey123456' | base64

Replace 'mysecretkey123456' with your actual key. This will output a base64-encoded string that you can use in the encryption configuration file.

Step 5: Configuring the Kubernetes API Server

The next step is to instruct the Kubernetes API server to use the newly created encryption configuration file. The API server's configuration is defined in the following file:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

This file is responsible for creating the pod for the API server. Add the following line under the - command: section:

- --encryption-provider-config=/etc/kubernetes/etcd-enc/etcd-encryption.yaml

This line tells the API server to use the encryption configuration file we created.

Step 6: Restart the API Server

After saving the changes to kube-apiserver.yaml, the API server will automatically restart with the new configuration. However, if you run the command:

kubectl get nodes

You will see an error. This occurs because the encryption configuration file exists on the host, but the API server container running inside a Kubernetes pod cannot access it.

Step 7: Mounting the Encryption Configuration File into the API Server Pod

To make the encryption configuration file accessible to the API server, mount it into the container. Open the kube-apiserver.yaml file again:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the following under the volumeMounts: section:

- mountPath: /etc/kubernetes/etcd-enc
  name: etc-kubernetes-etcd-enc
  readOnly: true

And add the following under the volumes: section:

- hostPath:
    path: /etc/kubernetes/etcd-enc
    type: DirectoryOrCreate
  name: etc-kubernetes-etcd-enc

Explanation of the Volume Mount Configuration

• mountPath:: Specifies the path inside the container where the encryption configuration file will be mounted.

• name:: The name of the volume mount, which must match between volumeMounts and volumes.

• hostPath:: Specifies the path on the host machine where the encryption configuration file is located.

• type: DirectoryOrCreate: Ensures that the directory is created if it does not exist.

Step 8: Verifying the Changes

After saving and exiting the file, wait for a few moments while the API server restarts. Now, try to create a secret:

kubectl create secret generic test-secret4 --from-literal=key=value

Then, check the etcd datastore again:

• For a standard Kubernetes setup:

    sudo vi /var/lib/etcd/member/snap/db
  

• For a Minikube setup:

    sudo vi /var/lib/minikube/etcd/member/snap/db
  

Try searching for the key or value again:

/keyOrval

This time, you should not find the secret stored in plain text, indicating that encryption at rest is working correctly!

Step 9: Troubleshooting the API Server

If the API server pod fails to start or if you encounter any errors, you can check the logs for detailed error messages. The logs can be found at:

cd /var/log/pods

Search for the correct pod directory for the API server. In a Minikube setup, it may look like:

cd kube-system_kube-apiserver-minikube_28a66b6eee103f0471478968cc9fe8c8/

Navigate into the directory:

cd kube-apiserver

Then, you can view the logs:

cat 1.log

If the path is different on your setup, the logs might be located at:

/var/log/kube-apiserver.log

Analyze the logs to find and fix any issues.

Conclusion

Encrypting Kubernetes secrets at rest is an essential security measure to protect sensitive data from unauthorized access. By following this guide, you have successfully configured encryption for your Kubernetes secrets, ensuring they are stored securely within etcd. Remember to monitor and manage encryption keys carefully, and regularly check logs for any issues or errors to maintain a secure and stable Kubernetes environment.