Deployment & Infra
Secrets at deploy: where they enter, where they leak
A teammate commits secret.yaml to the GitOps repo. It looks safe — the password field is cGFzc3dvcmQxMjM=, an opaque-looking blob. Six months later a security audit runs base64 -d on it and reads password123 in under a second. The repo is private, but it has 40 contributors, three forks, and a full git history that no force-push can scrub. The “encrypted” secret was never encrypted. base64 is encoding, not a lock.
The cardinal rule: never bake secrets into the image
The first failure happens before deploy. A Dockerfile with ENV API_KEY=sk-live-... or a COPY .env . bakes the secret into an image layer. That layer is content-addressed, cached, and pushed to a registry — and docker history plus docker save will replay every layer’s contents. Deleting the secret in a later layer does not remove it; the earlier layer still exists and is pullable by anyone with registry read access. The same applies to build args: --build-arg TOKEN=... lands in image metadata.
The rule a senior never breaks: secrets enter at deploy or runtime, not at build time. The image is a public-ish artifact — treat it as if it will leak. Configuration that changes per environment (dev, staging, prod) and anything secret is injected when the container starts, from outside the image. This is also why the same image promotes cleanly from staging to prod: it carries no environment inside it.
The Kubernetes Secret trap: base64 is not encryption
A Kubernetes Secret stores values base64-encoded. New engineers read the opaque string and assume it is protected. It is not. base64 is a reversible transport encoding — echo cGFzcw== | base64 -d reverses it in one second with zero key. Anyone who can read the Secret object, or who gets a copy of the manifest, reads the plaintext. The threat model is wide: a leaked YAML in a ticket, a backup, a misconfigured RBAC role, or the git history of a GitOps repo.
Worse, by default the value sits unencrypted in etcd, the cluster’s key-value store. Anyone with etcd access — a node compromise, a stolen backup — reads every secret in the cluster as plaintext. Encryption at rest is not on by default; you must explicitly configure an EncryptionConfiguration so the API server encrypts before writing to etcd. Even then, there is a trap: enabling it does not re-encrypt existing secrets. You must rewrite every Secret (e.g. kubectl get secrets -A -o json | kubectl replace -f -) after turning it on, or old values stay in the clear. Production setups back the encryption with a KMS provider (AWS KMS, GCP KMS) so the key lives outside the cluster.
| Belief | Reality | Consequence |
|---|---|---|
| ”base64 hides the value” | Reversible in <1s, no key | A leaked manifest = a leaked password |
| ”etcd encrypts my Secrets” | Plaintext by default | etcd backup theft = full secret dump |
| ”I turned on encryption” | Old Secrets not re-encrypted | Pre-existing values stay in the clear |
| ”Env vars are fine” | Leak via crash dumps + /proc | Secrets indexed in error trackers |
Env vars vs file mounts: the injection style matters
Once a secret is outside the image, you inject it two ways, and the choice has real security weight. Environment variables are simple — process.env.DB_PASSWORD — but they leak. They are inherited by every child process you spawn, they are readable at /proc/<pid>/environ by anything with the right access, and the killer one: a crash dump captures the full environment. Observability SDKs (Sentry, Datadog) routinely serialize process state on an error, so a stack trace ends up with your live API key indexed in plaintext in a third-party log store. Many app frameworks also print the environment on a fatal error page.
Mounted files are the safer default. The secret lands as a file (e.g. /run/secrets/db-password, mode 0400, on a tmpfs that never hits disk). It is not in the environment, not inherited by children, and not in /proc/.../environ. The decisive operational win: file mounts update in place when the secret rotates, while env vars are frozen at container start and need a pod restart to pick up a new value. For a credential you rotate every 30–90 days, that difference is the gap between a rolling restart and a silent in-place swap.
Why this works
Env vars feel safe because they are invisible in the code. But “invisible to me” is not “invisible to an attacker.” The exact properties that make env vars convenient — global to the process, inherited by children, dumped on crash — are the properties that leak them. A file you read once and never re-export has a far smaller blast radius.
The real toolchain: pull from a manager, encrypt before commit
Hand-managed Secrets do not scale. The production answer is to keep the source of truth in a dedicated secrets manager (HashiCorp Vault, AWS/GCP Secrets Manager) and pull from it at deploy/runtime:
- External Secrets Operator (ESO) — a controller that watches an external manager and syncs values into native Kubernetes Secrets on a
refreshInterval. Your manifests reference anExternalSecret, never the raw value. - Secrets Store CSI Driver — mounts secrets from the manager directly into the pod as files, bypassing etcd entirely; the value lives only in the pod’s tmpfs.
- Sealed Secrets — for GitOps where you must commit something: it asymmetrically encrypts the secret so only the in-cluster controller can decrypt. Now the blob in git is genuinely safe to commit, unlike a base64 Secret.
- Vault dynamic secrets — Vault generates a short-lived, per-app credential on demand (a DB user that exists for an hour, then auto-revokes). Nothing long-lived to leak; a stolen credential expires before it is useful.
A GitOps team must commit secret config to a private repo so Argo CD can apply it. Pick the approach.
A Kubernetes Secret's data field shows cGFzc3dvcmQ=. How protected is the value?
Why are mounted secret files often preferred over environment variables in production?
Order the secrets-handling decisions a senior makes, safest-first:
- 1 Never bake the secret into the image — no ENV, no COPY .env, no --build-arg
- 2 Keep the source of truth in a secrets manager (Vault / cloud Secrets Manager)
- 3 Pull at deploy/runtime via ESO or the CSI driver (or seal it for GitOps)
- 4 Prefer file mounts over env vars to shrink the leak surface
- 5 Rotate on a schedule; dynamic short-lived creds remove long-lived secrets entirely
- 01A colleague says 'our Kubernetes Secrets are safe because the values are base64-encoded.' Correct them precisely.
- 02Why do many teams inject secrets as mounted files rather than environment variables, and what does that buy them?
Getting secrets into a deployed app safely starts with one rule: they enter at deploy or runtime, never baked into the image, because image layers are cached, pushed to a registry, and replayable with docker history — a secret deleted in a later layer still lives in an earlier one. The Kubernetes Secret trap is that base64 is encoding, not encryption: a leaked manifest is a leaked password, and by default the value sits in etcd as plaintext, so you must explicitly enable encryption at rest (KMS-backed) and re-encrypt the existing Secrets afterward. Injection style matters too — environment variables leak through child processes, /proc, and crash dumps that get indexed by error trackers, while mounted files shrink that surface and, crucially, rotate in place without a restart. At scale the answer is a real toolchain: keep the source of truth in Vault or a cloud Secrets Manager, pull at runtime with the External Secrets Operator or the CSI driver, encrypt-before-commit with Sealed Secrets for GitOps, and prefer short-lived dynamic credentials so a stolen secret expires before it is useful.