Sealing your Secrets

Exploring the catalog Pod

The catalog deployment in the catalog Namespace accesses the following database values from the catalog-db secret via environment variables:

• DB_USER
• DB_PASSWORD

~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'

- name: DB_USER

  valueFrom:

    secretKeyRef:

      key: username

      name: catalog-db

- name: DB_PASSWORD

  valueFrom:

    secretKeyRef:

      key: password

      name: catalog-db

- name: DB_NAME

  valueFrom:

    configMapKeyRef:

      key: name

      name: catalog

- name: DB_READ_ENDPOINT

  valueFrom:

    secretKeyRef:

      key: endpoint

      name: catalog-db

- name: DB_ENDPOINT

  valueFrom:

    secretKeyRef:

      key: endpoint

      name: catalog-db

Upon exploring the catalog-db Secret we can see that it is only encoded with base64 which can be easily decoded as follows hence making it difficult for the secrets manifests to be part of the GitOps workflow.

~/environment/eks-workshop/base-application/catalog/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: catalog-db
data:
  username: "Y2F0YWxvZ191c2Vy"
  password: "ZGVmYXVsdF9wYXNzd29yZA=="

~$kubectl -n catalog get secrets catalog-db --template {{.data.username}} | base64 -d

catalog_user%

~$kubectl -n catalog get secrets catalog-db --template {{.data.password}} | base64 -d

default_password%

Let's create a new secret catalog-sealed-db. We'll create a new file new-catalog-db.yaml with the same keys and values as the catalog-db Secret.

~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml

apiVersion: v1
kind: Secret
metadata:
  name: catalog-sealed-db
  namespace: catalog
type: Opaque
data:
  password: ZGVmYXVsdF9wYXNzd29yZA==
  username: Y2F0YWxvZ191c2Vy

Now, let’s create SealedSecret YAML manifests with kubeseal.

~$kubeseal --format=yaml < ~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml \

> /tmp/sealed-catalog-db.yaml

Alternatively, the public key can be fetched from the controller and use it offline to seal your Secrets:

~$kubeseal --fetch-cert > /tmp/public-key-cert.pem

~$kubeseal --cert=/tmp/public-key-cert.pem --format=yaml < ~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml \

> /tmp/sealed-catalog-db.yaml

It will create a sealed-secret with the following content:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: catalog-sealed-db
  namespace: catalog
spec:
  encryptedData:
    password: AgBe(...)R91c
    username: AgBu(...)Ykc=
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: catalog-sealed-db
      namespace: catalog
    type: Opaque

Let's deploy the SealedSecret to your EKS cluster:

~$kubectl apply -f /tmp/sealed-catalog-db.yaml

The controller logs shows that it picks up the SealedSecret custom resource that was just deployed, unseals it to create a regular Secret.

~$kubectl logs deployments/sealed-secrets-controller -n kube-system

2022/11/07 04:28:27 Updating catalog/catalog-sealed-db

2022/11/07 04:28:27 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"catalog", Name:"catalog-sealed-db", UID:"a2ae3aef-f475-40e9-918c-697cd8cfc67d", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"23351", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

Verify that the catalog-sealed-db Secret unsealed from the SealedSecret was deployed by the controller to the secure-secrets namespace.

~$kubectl get secret -n catalog catalog-sealed-db

NAME                       TYPE     DATA   AGE

catalog-sealed-db          Opaque   4      7m51s

Let's redeploy the catalog deployment that reads from the above Secret. We have updated the catalog deployment to read the catalog-sealed-db Secret as follows:

Kustomize Patch

~/environment/eks-workshop/modules/security/sealed-secrets/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
spec:
  template:
    spec:
      containers:
        - name: catalog
          env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: catalog-sealed-db
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: catalog-sealed-db

Deployment/catalog

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/created-by: eks-workshop
    app.kubernetes.io/type: app
  name: catalog
  namespace: catalog
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: service
      app.kubernetes.io/instance: catalog
      app.kubernetes.io/name: catalog
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/component: service
        app.kubernetes.io/created-by: eks-workshop
        app.kubernetes.io/instance: catalog
        app.kubernetes.io/name: catalog
    spec:
      containers:
        - env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: catalog-sealed-db
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: catalog-sealed-db
          envFrom:
            - configMapRef:
                name: catalog
          image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 3
          name: catalog
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            successThreshold: 3
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 512Mi
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          volumeMounts:
            - mountPath: /tmp
              name: tmp-volume
      securityContext:
        fsGroup: 1000
      serviceAccountName: catalog
      volumes:
        - emptyDir:
            medium: Memory
          name: tmp-volume

Diff

             - name: DB_USER
               valueFrom:
                 secretKeyRef:
                   key: username
-                  name: catalog-db
+                  name: catalog-sealed-db
             - name: DB_PASSWORD
               valueFrom:
                 secretKeyRef:
                   key: password
-                  name: catalog-db
+                  name: catalog-sealed-db
           envFrom:
             - configMapRef:
                 name: catalog
           image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0

~$kubectl apply -k ~/environment/eks-workshop/modules/security/sealed-secrets

~$kubectl rollout status -n catalog deployment/catalog --timeout 30s

The catalog-sealed-db which is a SealedSecret resource is safe to be stored in a Git repository along with YAML manifests pertaining to other Kubernetes resources such as DaemonSets, Deployments, ConfigMaps etc. deployed in the cluster. You can then use a GitOps workflow to manage the deployment of these resources to your cluster.