Kubernetes Secrets Management — SOPS + age (GitOps‑friendly)
Owner: Platform Engineering
Audience: Devs & Ops
Status: Adopted
This document captures the exact process we use to manage Kubernetes secrets with Mozilla SOPS and the age encryption tool. It includes why we chose this approach, how to work with secrets day‑to‑day, and trade‑offs versus other methods.
Executive Summary
We store Kubernetes Secret manifests in Git encrypted at rest using SOPS with an age public key. SOPS encrypts your secret values with AES‑256‑GCM, which is industry gold standard. In SOPS you can not swap AES‑256‑GCM out. The age private key lives only in protected CI variables. CI decrypts the manifests at deploy time and applies them with kubectl. No in‑cluster controllers, CRDs, or webhooks are required.
Why this approach
Top benefits:
• Zero in‑cluster installs (no CRDs/controllers to run or break).
• GitOps‑native: encrypted YAML sits in the repo; diffs remain readable (metadata & keys stay visible).
• Cloud‑agnostic: age works anywhere; no AWS/GCP KMS dependency.
• Simple local edits: devs can encrypt/decrypt locally with one key pair and a small sops.yaml rule.
• Least‑privilege: only CI needs the private key; developers can commit without decrypting.
Alternatives at a glance
Components
• SOPS: YAML/JSON editor that transparently encrypts selected fields.
• age: modern, simple encryption (public key in repo; private key in CI variable SOPS_AGE_KEY).
• Repo layout: infra/secrets/*.secret.yaml committed encrypted.
• sops.yaml: creation rules that match files to the age public key.
• CI job: decrypts and kubectl apply -f - per file.
Create the age key file (agekey.txt)
Generate a new keypair locally. Keep the file private and never commit it to Git.
bash
# Generate keypair and write to agekey.txt (private)
age-keygen -o agekey.txt
# Show the public key (for sops.yaml)
sed -n 's/^# public key: //p' agekey.txt
# export the private key into CI protected variable SOPS_AGE_KEY
# (copy the single line that begins with AGE-SECRET-KEY-1)
Example contents of agekey.txt (sanitized):
# created: 2025-08-23T11:22:33Z
# public key: age1EXAMPLEPUBLICKEYxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AGE-SECRET-KEY-1EXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLEEXAMPLE12345
Place the public key string (the age1...) into your sops.yaml under creation_rules[].age. Place the private key line (AGE-SECRET-KEY-1...) into your CI system's protected variable SOPS_AGE_KEY.
Configuration — sops.yaml
Minimal example (public key redacted):
creation_rules:
- path_regex: secrets/.*\.secret\.yaml$
age: "age1...REDACTED...cpndg4"
Authoring a Secret manifest
Create a normal Kubernetes Secret then encrypt with SOPS.
apiVersion: v1
kind: Secret
metadata:
name: auth-db
namespace: example
type: Opaque
stringData:
PGHOST: "db"
PGPORT: "5432"
PGDATABASE: "auth-service-db"
PGUSER: "authsvc"
PGPASSWORD: "supersecret" # <-- will be encrypted by SOPS
Encrypted Secret manifest in Git after SOPS
How‑to: Encrypt, decrypt, and edit
Encrypt a new/updated secret
bash
sops --encrypt --config sops.yaml --in-place secrets/auth-db.secret.yaml
Decrypt, change values, re‑encrypt
bash
sops --decrypt auth-db.secret.yam
# copy and edit in place
sops --encrypt --config sops.yaml --in-place secrets/auth-db.secret.yaml
Tip: You can also use sops auth-db.secret.yaml to open an inline editor and it will handle decrypt+encrypt automatically on save.
CI/CD: Applying secrets safely
We use a tiny image with sops + kubectl. CI holds the private age key and the kubeconfig as protected variables. The job decrypts each file and streams it to kubectl.
Container image (sops + kubectl)
FROM alpine:3.20
RUN apk add --no-cache curl bash ca-certificates \
&& curl -sL https://github.com/mozilla/sops/releases/download/v3.9.0/sops-v3.9.0.linux.amd64 -o /usr/local/bin/sops \
&& chmod +x /usr/local/bin/sops \
&& curl -sL https://dl.k8s.io/release/v1.30.2/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
&& chmod +x /usr/local/bin/kubectl
CI variables (protected)
• KUBECONFIG_B64: base64‑encoded kubeconfig for the target cluster.
• SOPS_AGE_KEY: the private age key (contents of the key file, not the public key).
GitLab CI variables: KUBECONFIG_B64 and SOPS_AGE_KEY
GitLab job: decrypt and apply
deploy:apply-secrets:
stage: example
image: "${CI_REGISTRY_IMAGE}/sops-kubectl:latest"
rules:
- when: manual
script:
# 1) Write kubeconfig from CI var
if [ -n "${KUBECONFIG_B64}" ]; then
echo "${KUBECONFIG_B64}" | base64 -d > /tmp/kubeconfig
export KUBECONFIG=/tmp/kubeconfig
else
echo "ERROR: set KUBECONFIG_B64" >&2; exit 1
fi
# 2) Provide private age key to SOPS
if [ -z "${SOPS_AGE_KEY}" ]; then
echo "ERROR: SOPS_AGE_KEY not set" >&2; exit 1
fi
# 3) Decrypt and apply each secret
set -euo pipefail
for f in infra/secrets/*.secret.yaml; do
echo "Applying $f"
sops --decrypt "$f" | kubectl apply -f -
done
Manual 'apply-secrets' job in the pipeline
Operations & Key Management
• Generate keys: age-keygen -o age.key (keep private). Public key is the line beginning with age1....
• Where keys live: public in sops.yaml; private only in CI protected variable SOPS_AGE_KEY.
• Rotation: generate a new key; add it to sops.yaml; re-encrypt files (sops -r -i); update CI variable; remove old key.
• Access: Only a small set of maintainers can trigger the manual job; CI logs never print plaintext.
• Backups/DR: store the private key in an offline secret manager or printed escrow, with dual control.
Advantages
• No CRDs or controllers in cluster; fewer moving parts and blast radius.
• Works in any environment (local, on‑prem, cloud) identically.
• Auditable Git history; encrypted diffs still meaningful.
• Fast developer workflow; minimal tooling to install locally.
Disadvantages / Constraints
• Key management is on us: must protect/rotate the age private key and CI access.
• No built‑in dynamic/short‑lived secrets like Vault or ESO can provide.
• Secrets materializes in the CI runner memory during kubectl apply.
• If the private key is lost, re‑encryption with a new key is required from a copy that still decrypts.
Appendix: Common commands
# Generate a new age keypair
age-keygen -o age.key
# Show public key (starts with age1...)
grep -E 'public key' -n age.key || cat age.key | sed -n 's/^# public key: //p'
# Re-encrypt all files after changing keys
sops --encrypt --in-place --config sops.yaml infra/secrets/*.secret.yaml
# Decrypt one file to stdout
sops --decrypt infra/secrets/auth-db.secret.yaml
Comments
Post a Comment