How‑To: Nightly Postgres → MinIO Backups with Kubernetes CronJob (plus lifecycle)

Updated: 2025-08-24

Overview

Create a CronJob that runs `pg_dumpall`, compresses the dump, and uploads to MinIO. Add a lifecycle rule to age off old backups automatically.

Prerequisites

- Postgres reachable from the cluster
- MinIO endpoint and access keys
- kubectl access to create Secrets and CronJobs

Secrets (example):

kubectl -n backups create secret generic minio-env   --from-literal=MC_HOST_minio="http://ACCESSKEY:SECRETKEY@minio.minio.svc.cluster.local:9000"   --from-literal=BUCKET="pg-backups"

kubectl -n backups create secret generic pg-env   --from-literal=PGHOST=postgres.postgres.svc.cluster.local   --from-literal=PGPORT=5432   --from-literal=PGUSER=postgres

CronJob manifest:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: pg-backup
  namespace: backups
spec:
  schedule: "0 3 * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: backup
            image: alpine:3.20
            envFrom:
            - secretRef: { name: minio-env }
            - secretRef: { name: pg-env }
            command: ["/bin/sh","-lc"]
            args:
            - |
              set -euo pipefail
              apk add --no-cache postgresql-client minio-client
              ts=$(date -u +%Y%m%dT%H%M%SZ)
              pg_dumpall -h "$PGHOST" -p "$PGPORT" -U "$PGUSER" -f /tmp/cluster.sql
              gzip -9 /tmp/cluster.sql
              mc mb -p minio/"$BUCKET" || true
              mc cp /tmp/cluster.sql.gz minio/"$BUCKET"/cluster-$ts.sql.gz
            resources:
              requests: { cpu: "50m", memory: "128Mi", ephemeral-storage: "1Gi" }
              limits:   { cpu: "500m", memory: "512Mi", ephemeral-storage: "5Gi" }

Lifecycle policy (delete after 14 days):

# one-off job/pod to set ILM on the bucket
kubectl -n backups run --rm -it mc --image=alpine:3.20 -- sh -lc '
  apk add --no-cache minio-client &&
  mc alias set minio "$MC_HOST_minio" &&
  mc ilm add --expiry-days 14 minio/pg-backups &&
  mc ilm ls minio/pg-backups'

Restore (Quick):

# Download a backup and restore all DBs
mc cp minio/pg-backups/cluster-<timestamp>.sql.gz .
gunzip cluster-<timestamp>.sql.gz
psql -h $PGHOST -p $PGPORT -U postgres -f cluster-<timestamp>.sql

Verification

kubectl -n backups get cronjob pg-backup
kubectl -n backups logs job/pg-backup-<timestamp>
# List objects:
kubectl -n backups run --rm -it mc --image=alpine:3.20 -- sh -lc '
  apk add --no-cache minio-client &&
  mc alias set minio "$MC_HOST_minio" &&
  mc ls minio/pg-backups'

Troubleshooting

- `/bin/sh: MINIO_ACCESS_KEY: parameter not set` → use MC_HOST_minio, not old env var names.
- `AccessDenied` → wrong ACCESS/SECRET or bucket policy; test with `mc alias set`.
- Pod OOM/ephemeral storage errors → increase resource limits; run off-peak.
- Network/TLS issues → verify service DNS names and CA trust to MinIO.

Security

- Use a write-only MinIO key scoped to the backup bucket.
- Enable SSE (server-side encryption) or KMS.
- Keep DB creds in Secrets; avoid storing in images/logs.
- Consider PITR with WAL archiving for production RPO < 24h.

Meme idea

Backups are like umbrellas—you miss them most when the sky opens up.

Taylor Swift

“I remember it all too well.” — All Too Well