Managing the Sealing Key
The only way to decrypt the encrypted data within a SealedSecret is with the sealing key that is managed by the controller. There could be situations where you are trying to restore the original state of a cluster after a disaster or you want to leverage GitOps workflow to deploy the Kubernetes resources, including SealedSecrets, from a Git repository and create a new EKS cluster. The controller deployed in the new EKS cluster must use the same sealing key to be able to unseal the SealedSecrets.
Run the following command to retrieve the sealing key from the cluster. In a production environment, it's considered a best practice to make use of Kubernetes RBAC to grant the permissions required to perform this operation to restricted set of clients.
In order to test the operation let’s delete the Secret which contains the sealing key and recycle the sealed secrets controller:
Now, we'll check the logs for the controller. Notice that it fails to decrypt the SealedSecret:
[...]
2022/11/18 22:47:42 Updating catalog/catalog-sealed-db
2022/11/18 22:47:43 Error updating catalog/catalog-sealed-db, giving up: no key could decrypt secret (password, username, endpoint, name)
E1118 22:47:43.030178 1 controller.go:175] no key could decrypt secret (password, username, endpoint, name)
2022/11/18 22:47:43 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"catalog", Name:"catalog-sealed-db", UID:"a6705e6f-72a1-43f5-8c0b-4f45b9b6f5fb", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"519192", FieldPath:""}): type: 'Warning' reason: 'ErrUnsealFailed' Failed to unseal: no key could decrypt secret (password, username, endpoint, name)This is because we deleted the sealing key, which caused the controller to generate a new one when it started. This effectively makes all of our SealedSecret resources inaccessible by this controller. Thankfully we previously saved it to /tmp/master-sealing-key.yaml so we can re-create it in the EKS cluster:
Checking the logs again shows this time the controller picked up the sealing key we restored and unsealed our catalog-sealed-db secret:
[...]
2022/11/18 22:52:51 Updating catalog/catalog-sealed-db
2022/11/18 22:52:51 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"catalog", Name:"catalog-sealed-db", UID:"a6705e6f-72a1-43f5-8c0b-4f45b9b6f5fb", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"519192", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfullyThe file /tmp/master-sealing-key.yaml contains the public/private key pair generated by the controller. If this file is compromised all the SealedSecret manifests can be unsealed and the encrypted sensitive information they store revealed. As such this file must be guarded by granting least privilege access. For additional guidance on topics such as sealing key renewal and manual sealing key management please consult the documentation.
One option to secure the sealing key is to store the /tmp/master-sealing-key.yaml file contents as a SecureString parameter in AWS Systems Manager Parameter Store. The parameter could be secured using a KMS Customer managed key (CMK) and you can use the Key policy to restrict the set of IAM principals who can use this key in order to retrieve the parameter. Additionally you may also enable automatic rotation of this CMK in KMS. Note that standard tier parameters support a maximum parameter value of 4096 characters. Hence, given the size of the master.yaml file, you will have to store it as a parameter in the Advanced tier.