Skip to main content

Mounting AWS Secrets Manager secret on Kubernetes Pod

Now that we have a secret stored in AWS Secrets Manager and synchronized with a Kubernetes Secret let's mount it inside the Pod. First we should take a look at the catalog Deployment and the existing Secrets in the catalog namespace.

The catalog Deployment accesses the following database credentials from the catalog-db secret via environment variables:

  • DB_USER
  • DB_PASSWORD
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'
- name: DB_USER
  valueFrom:
    secretKeyRef:
      key: username
      name: catalog-db
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: catalog-db
Notice that the catalog Deployment doesn't have any volumes or volumeMounts other than an emptyDir mounted on /tmp
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.volumes'
- emptyDir:
    medium: Memory
  name: tmp-volume
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .volumeMounts'
- mountPath: /tmp
  name: tmp-volume
Let's go and apply the changes in the catalog Deployment to use the secret stored in AWS Secrets Manager as the source of the credentials.
~/environment/eks-workshop/modules/security/secrets-manager/mounting-secrets/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base-application/catalog
patches:
  - path: deployment.yaml
  - path: serviceaccount.yaml
Here we will be mounting the AWS Secrets Manager secret using the CSI driver with the SecretProviderClass we validated earlier on the /etc/catalog-secret mountPath inside the Pod. When this happens, AWS Secrets Manager will sync the contents of the stored secret with Amazon EKS, and create a Kubernetes Secret with the same contents that can be consumed as environment variables in the Pod.
~$kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/mounting-secrets/ \
  | envsubst | kubectl apply -f-
~$kubectl rollout status -n catalog deployment/catalog --timeout=120s
Let's validate the changes made in the catalog Namespace.
In the Deployment, we'll be able to check that it has a new volume and respective volumeMount pointing to the CSI Secret Store Driver, and mounted on /etc/catalog-secrets path.
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.volumes'
- csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: catalog-spc
  name: catalog-secret
- emptyDir:
    medium: Memory
  name: tmp-volume
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .volumeMounts'
- mountPath: /etc/catalog-secret
  name: catalog-secret
  readOnly: true
- mountPath: /tmp
  name: tmp-volume
Mounted Secrets are a good way to have sensitive information available as a file inside the filesystem of one or more of the Pod's containers. Some benefits are not exposing the value of the secret as environment variables and when a volume contains data from a Secret, and when that Secret is updated Kubernetes tracks it and updates the data in the volume.
You can take a look on the contents of the mounted Secret inside your Pod.
~$kubectl -n catalog exec deployment/catalog -- ls /etc/catalog-secret/
eks-workshop-catalog-secret  password  username
~$kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/${SECRET_NAME}
{"username":"catalog_user", "password":"default_password"}
~$kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/username
catalog_user
~$kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/password
default_password
There are 3 files in the mountPath /etc/catalog-secret. `
  1. eks-workshop-catalog-secret: The value of the secret in JSON format.
  2. password: password jmesPath filtered and formatted value.
  3. username: username jmesPath filtered and formatted value.
Also, the environment variables are now being consumed from the new Secret, catalog-secret that didn't exist earlier, and it was created by the SecretProviderClass via CSI Secret Store driver.
~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'
- name: DB_USER
  valueFrom:
    secretKeyRef:
      key: username
      name: catalog-secret
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: catalog-secret
~$kubectl -n catalog get secrets
NAME             TYPE     DATA   AGE
catalog-db       Opaque   2      15h
catalog-secret   Opaque   2      43s
We can verify this by checking the environment variables in the running pod:
~$kubectl -n catalog exec -ti deployment/catalog -- env | grep DB_
Now we have a Kubernetes Secret fully integrated with AWS Secrets Manager that can leverage secret rotation, which is a best practice for Secrets Management. Every time a secret is rotated or updated on AWS Secrets Manager, we can roll out a new version of your Deployment so the CSI Secret Store driver can sync the Kubernetes Secrets contents with the rotated value.