Secrets in Kubernetes
A Kubernetes Secret is base64, not encryption — plaintext to anyone with etcd, a node, or RBAC read. Why that matters and how to manage secrets without copies sprawling across namespaces and CI logs.
A junior on your team is debugging a crashing pod and pastes a manifest into the ticket: kubectl get secret db-creds -o yaml. The data: field is a wall of base64, so they think it’s safe to share — it looks encrypted. You decode one line in your head: cG9zdGdyZXM6Ly9hZG1pbjpodW50ZXIyQA== is just postgres://admin:hunter2@. The production database password is now in a Jira comment, indexed by search, visible to every contractor with view access. No CVE, no exploit. Just base64 mistaken for a lock.
By the end of this lesson you’ll know exactly why a Kubernetes Secret is not encrypted, who can read it, and how to manage secrets without the sprawl that turns one leak into a fleet-wide compromise.
base64 is encoding, not encryption
This is the most expensive misconception in Kubernetes security, so let me say it plainly: base64 is a reversible, keyless transcoding. Anyone, anywhere, can run base64 -d and get the original bytes back. No secret material is involved in decoding — that’s the whole point of base64: it exists so binary data survives a text-only channel, not to protect it. Calling a Secret “encrypted” because the YAML shows aHVudGVyMg== instead of hunter2 is like calling a window “locked” because you taped paper over it.
What a Kubernetes Secret actually buys you over a ConfigMap is operational, not cryptographic: it’s flagged so it isn’t printed in plain logs by default; it can be mounted as a tmpfs (in-memory) volume rather than written to disk; access is governed through RBAC as a distinct resource type. Those are real, useful properties. None of them is confidentiality on disk. The bytes sit in etcd, and by default they sit there in plaintext.
Who can actually read your Secret
The mature question is never “is it base64?” but “what is the read surface?”. A Secret is exposed to a surprisingly wide blast radius, and each path is a real attack we’ve seen in postmortems:
| Read path | Who gets access | What stops it |
|---|---|---|
| etcd on disk | Anyone with a node, a backup, or a stolen etcd snapshot | EncryptionConfiguration (KMS provider), not base64 |
RBAC get/list secrets | Any user/SA with that verb in the namespace | Least-privilege RBAC; never grant list secrets broadly |
| Mounted into a pod | Anyone who can exec into the pod or read its FS | Scope the SA; mount only what the workload needs |
| Default ServiceAccount token | A compromised pod reaching the API with auto-mounted rights | automountServiceAccountToken: false |
| CI logs / Git | Anyone reading a pipeline log or committing the manifest | External secret store; never commit a rendered Secret |
Two paths surprise people the most: etcd on disk and RBAC list secrets. By default kube-apiserver writes Secret objects to etcd unencrypted — so an etcd backup in an S3 bucket, or a stolen volume snapshot, is a full credential dump. And the list verb on secrets in RBAC is catastrophic precisely because it returns contents, not just names: a single over-broad Role with list on secrets in a namespace hands every credential in that namespace to whoever holds it. It’s near the top of the OWASP Kubernetes Top 10 for a reason.
Encryption at rest: turning the lock on
The fix for the etcd path is encryption at rest, configured on the API server via EncryptionConfiguration. The lesson here is which provider you pick, because the obvious option is a trap.
The trap is the aescbc/aesgcm providers with the key in the config file: yes, they encrypt etcd, but the encryption key sits on the control-plane node in plaintext, so an attacker who can read etcd’s data can usually read the key file too. That’s a sideways step, not a fix. The strong choice is a KMS provider performing envelope encryption: each secret is encrypted with a data-encryption key (DEK), the DEK is encrypted with a key-encryption key (KEK) that lives only inside an external KMS (AWS KMS, GCP KMS, Vault), and the API server must make a network call to the KMS to unwrap it. Now a stolen etcd snapshot is inert ciphertext. Note one catch: turning on encryption only encrypts secrets written after you enable it — you must rewrite every existing secret (kubectl get secrets -A -o json | kubectl replace -f -) to actually seal what’s already there.
▸Why this works
Why can’t you just turn on encryption at rest and call it done? Because encryption at rest protects only one of the five read paths in the table — the stolen-etcd path. It does nothing against over-broad RBAC list, against kubectl exec into a pod with the secret mounted, or against a credential pasted into a CI log. Encryption at rest is necessary and frequently skipped, but it’s a single layer. The read surface is what you actually manage, and most real secret leaks happen through RBAC and human copy-paste, not stolen disks.
Secret sprawl: the real operational failure
Even with encryption at rest enabled and RBAC tight, the failure that actually bites teams is sprawl: the same secret copied into ten namespaces, baked into three Docker images, pasted into two CI variable stores, and committed once to Git “just to unblock the deploy.” Sprawl is what turns one rotation into a week of archaeology, and one leak into a fleet-wide incident, because you can no longer answer “where does this credential live?”.
The mature pattern is a single source of truth outside the cluster plus sync-in, not copy-in. You hold the canonical secret in a dedicated store (Vault, AWS Secrets Manager, GCP Secret Manager), and a controller — the External Secrets Operator (ESO) or the Secrets Store CSI Driver — pulls it into the cluster as a native Secret on demand. Manifests reference an ExternalSecret pointing at the store; the credential itself never sits in Git, never in an image, and rotation happens in one place and propagates. The in-cluster Secret becomes a cache, not a source-of-record database.
Scope a Secret read to one object
1/3You have one DB password used by services across six namespaces. Today it's a hand-created Secret copied into each. Pick the approach that best prevents sprawl and makes rotation a single-place operation.
A colleague says: 'our secrets are safe in etcd because Kubernetes base64-encodes them.' What's the precise correction?
You enable encryption at rest with a KMS provider on a cluster that already has 200 secrets. What else must you do, and why?
Order these from weakest to strongest as a way to protect a DB password in Kubernetes:
- 1 base64 Secret committed to Git (plaintext in history)
- 2 Plain Secret in etcd, no encryption at rest
- 3 Encryption at rest with a local aescbc key on the node
- 4 Encryption at rest via an external KMS (envelope encryption)
- 5 External store + ESO sync, tight RBAC, no copies in Git/images
- 01Explain why a Kubernetes Secret is not encrypted, and list who can actually read it.
- 02What is secret sprawl, why is it a real failure, and what pattern prevents it?
A Kubernetes Secret is base64-encoded, and that’s keyless encoding, not encryption — base64 -d reverses it instantly, so a secret in a Jira comment or a committed manifest is a plaintext credential leak. Compared to a ConfigMap, a Secret buys operational properties (not printed in plain logs, tmpfs mounting, separate RBAC) but never on-disk confidentiality. Its read surface is wide: plaintext in etcd by default, RBAC list (which returns contents), pod mounts, auto-mounted ServiceAccount tokens, and CI/Git. Closing the etcd path means enabling encryption at rest — and choosing a KMS provider for true envelope encryption, not a local aescbc key sitting on the same node, and remembering to rewrite existing secrets since encryption only applies on write. But the failure that actually causes incidents is sprawl: copies of one credential scattered across namespaces, images, CI, and Git, making rotation impossible and turning one leak into a fleet-wide compromise. The mature fix is one external source of truth synced in via the External Secrets Operator or CSI driver — a reference, not a copy — so the in-cluster Secret is a cache and rotation happens in exactly one place. Next time you see a data: field full of base64, your reflex should be: who can read etcd, who has list, and how many copies of this exist?
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.
Apply this
Put this lesson to work on a real build.