open atlas
↑ Back to track
AWS, hands-on AWS · 07 · 02

IAM evaluation deep dive, KMS envelope encryption, and Secrets Manager

IAM resolves in a fixed order: default deny, an explicit allow grants, an explicit deny wins; boundaries and SCPs cap the ceiling, conditions scope each statement. Prefer roles+STS over long-lived keys, stop the confused deputy with ExternalId, encrypt with KMS envelope keys.

AWS Senior ◷ 20 min
Level
FoundationsJuniorMiddleSenior

A developer commits a .env to a public repo for ninety seconds before force-pushing it away. Too late: bots scrape new GitHub commits within seconds, and the long-lived IAM access key inside is already harvested. Minutes later that key — attached to a user whose policy carried a convenient "Action": "*", "Resource": "*" — is spinning up dozens of GPU instances for crypto mining across every region, and exfiltrating an S3 bucket on the side. The bill crosses five figures before billing alerts even fire. The key was never rotated because it was never temporary, the policy was never scoped because a wildcard “just worked,” and nothing capped the blast radius because no permissions boundary or SCP existed. Three separate guardrails were missing, and the breach needed only one of them to be present. This lesson is about all three.

By the end you will understand exactly why the war story’s three missing guardrails each mattered, and how to put all three in place before the next credential leaks.

How a request is actually decided

You already met identity policies in the core IAM lesson: a principal, an action, a resource, an effect. The senior model is the evaluation — how AWS combines several policy types into one yes-or-no decision for a single API call. The starting point is never neutral: every request is an implicit deny by default. From there the rule has three moves, and the order is fixed.

First, an explicit allow in any applicable policy can flip the default to “allowed.” Second, the policy types are combined: identity-based and resource-based policies form a union (an allow in either one is enough within the same account), while permissions boundaries and Service Control Policies intersect — the request must be allowed by all applicable categories. Third, and overriding everything: an explicit Deny anywhere always wins. A deny in an identity policy, a resource policy, a boundary, or an SCP cannot be out-voted by any number of allows. That is the single most important sentence in IAM, and most “why can’t I access this?” tickets end at an explicit deny someone forgot about.

The policy types stack into layers, each playing a distinct role:

  • Identity-based policies — attached to a user, group, or role; grant what that principal may do.
  • Resource-based policies — attached to the resource (an S3 bucket policy, a KMS key policy, an SQS queue policy); grant who may touch that resource, and uniquely enable cross-account access.
  • Permissions boundaries — an advanced identity policy that sets the maximum permissions a principal can ever have. It grants nothing by itself; it caps. A principal’s effective permissions are the intersection of its identity policies and its boundary.
  • Service Control Policies (SCPs) — org-wide guardrails attached at the AWS Organizations level. They also grant nothing; they cap what every account and principal under them can do, no matter how permissive a local admin gets.
  • Session policies — passed inline when you assume a role; they further narrow the session’s permissions for its lifetime.

So effective permissions = (the union of allows), intersected with every applicable boundary and every applicable SCP, minus any explicit deny. The wildcard policy in the war story granted everything, but with no boundary and no SCP the intersection never shrank — the ceiling was infinite, so the leaked key inherited infinite reach. A PowerUserAccess boundary, or an SCP denying iam:* outside a break-glass role and denying actions outside approved regions, would have turned a catastrophe into a contained annoyance.

Conditions make least privilege real

A statement with "Resource": "*" is a liability; a statement scoped by a Condition is least privilege. Condition keys test the request context and gate the statement: aws:SourceIp (only from the office CIDR), aws:PrincipalOrgID (only principals in your org), aws:SecureTransport (only over TLS), aws:RequestedRegion, or StringEquals on a resource tag (only resources tagged with your team). A statement applies only when its conditions evaluate true. This policy lets a principal read one bucket, but only over TLS and only from a known network — drop either condition and the request is denied:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadReportsTlsAndOffice",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::acme-reports/*",
      "Condition": {
        "Bool": { "aws:SecureTransport": "true" },
        "IpAddress": { "aws:SourceIp": "203.0.113.0/24" }
      }
    }
  ]
}

The deeper habit conditions enable is roles over long-lived keys. A long-lived access key is a static, exportable, often-unrotated credential — exactly what leaked in the hook. An IAM role issues temporary credentials through AWS STS: short-lived, auto-expiring, never committed to a repo because they only exist for minutes. Every workload — an EC2 instance (via an instance profile), a Lambda, an ECS task, a CI job (via OIDC) — should assume a role, not carry a key. When you must audit what’s out there, IAM Access Analyzer flags resource policies that grant access to external accounts and surfaces unused permissions you can prune.

Cross-account access and the confused deputy

Cross-account access is built on roles: a role in account B has a trust policy naming account A as allowed to call sts:AssumeRole. Account A’s principals assume the role and act in B with B’s permissions. But naming an external account in a trust policy opens a classic hole — the confused deputy. Per AWS, this is “a security issue where an entity that doesn’t have permission to perform an action can coerce a more-privileged entity to perform the action.” If you hire a third-party SaaS that assumes a role in many customers’ accounts, and your role ARN isn’t a secret, another of their customers could hand your ARN to the vendor and trick the vendor (the “deputy”) into acting on your resources.

The fix is an ExternalId condition in the trust policy. The vendor generates a unique ID per customer and passes it on every AssumeRole call; your trust policy requires your ID, so a request carrying anyone else’s ID fails:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::222222222222:root" },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": { "sts:ExternalId": "a1b2c3-unique-per-customer" }
    }
  }
}

For AWS service principals doing cross-service work (CloudTrail writing to your bucket, for example), the analogous guards are aws:SourceArn and aws:SourceAccount conditions, which assert the service is acting on behalf of a resource or account you actually expect.

KMS: envelope encryption and key policies

Why does encrypting data at scale require two keys instead of one? Because you cannot stream a 5 GB object through a cloud API, and handing out the master key would make rotation a nightmare. Encryption at scale uses envelope encryption, and KMS is built around it. A KMS key (the customer-managed key) lives inside a hardware security module and is “designed never to be exported from the HSM in plaintext.” You never get the key’s bytes. Instead you call GenerateDataKey, and KMS returns a data key in two forms: a plaintext copy and a copy encrypted under your KMS key. You encrypt your large payload locally with the plaintext data key, immediately scrub the plaintext from memory, and store the encrypted data key alongside the ciphertext. To decrypt later, you send only the encrypted data key back to KMS, which unwraps it and returns the plaintext data key for that one operation. The bulk data never travels to KMS; only the small data key does. This is why S3, EBS, and RDS encryption scale — each object gets its own data key, all wrapped by one central KMS key.

Who may use and manage a KMS key is governed by its key policy — a resource-based policy that, unlike most others, is the primary access control for the key. A dangerous footgun: if a key policy does not grant access to your account’s root (or another administrative principal), you can lock everyone out of the key permanently — the data encrypted under it becomes unrecoverable, and not even AWS Support can override the policy. Beyond key policies, grants delegate temporary, narrowly-scoped use to a principal (often an AWS service) without editing the policy; aliases give a key a stable friendly name so you can rotate the underlying material without changing references; automatic rotation (optional for customer-managed keys, annual by default and now configurable) generates new backing key material while keeping old versions to decrypt old data; and keys are symmetric (one key encrypts and decrypts — the default, for most data) or asymmetric (a public/private pair, for signing or encrypting with a public key you can hand out). A KMS key policy looks like this:

{
  "Version": "2012-10-17",
  "Id": "key-policy-app-data",
  "Statement": [
    {
      "Sid": "EnableRootAccountAdmin",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AllowAppToEncryptDecrypt",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/app-runtime" },
      "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
      "Resource": "*",
      "Condition": {
        "StringEquals": { "kms:ViaService": "s3.us-east-1.amazonaws.com" }
      }
    }
  ]
}

Drop that first statement and you have orphaned the key.

LayerWhat it doesGrants or caps?Failure if missing or wrong
Identity policyWhat this principal may doGrantsWildcard grants everything; leaked key inherits it
Permissions boundaryMax a principal can ever haveCaps (intersection)No ceiling on the blast radius
SCPOrg-wide guardrailCaps (intersection)No org-level brake on any account
Explicit denyOverrides any allowKill switchRisky action stays allowed by default
KMS key policyWho uses/manages the keyGrants (resource-based)Omit account root and you orphan the key forever
Why this works

Why store the encrypted data key next to the data instead of just calling KMS to encrypt the payload directly? Because KMS limits the size of data you can pass it (a few KB) and every call costs a request — you cannot stream a 5 GB object through KMS. Envelope encryption inverts the flow: KMS only ever handles the tiny data key, your own CPU does the bulk symmetric encryption locally at line rate, and the wrapped data key rides along with the ciphertext so any holder of decrypt permission on the KMS key can recover it. One central, audited, rotatable key protects an unlimited volume of data.

Secrets Manager versus Parameter Store

The leaked .env is the deeper lesson: secrets do not belong in code, images, or environment files baked into a build. AWS Secrets Manager stores secrets encrypted with KMS and — its headline feature — supports automatic rotation. For database credentials it runs a Lambda rotation function on a schedule: the function creates a new credential, updates the database, and swaps staging labels (AWSPENDING becomes AWSCURRENT, the old one becomes AWSPREVIOUS) so callers always read the live value without an outage. It integrates directly with RDS, Redshift, and DocumentDB.

The cheaper, simpler sibling is SSM Parameter Store. Its SecureString parameters are also KMS-encrypted, it has no per-secret monthly charge, and it’s perfect for config plus secrets you don’t need to rotate automatically. The tradeoff: Parameter Store has no built-in rotation and no native database integration. The senior call: Secrets Manager for credentials that must rotate (databases, API keys with a lifecycle); Parameter Store for static config and low-churn secrets where cost matters.

Pick the best fit

You need to store the production database password your app reads on every cold start. Compliance requires it to rotate every 30 days with zero downtime. Pick the store.

Quiz

An identity policy allows s3:DeleteObject on a bucket, but an SCP on the account contains an explicit Deny for s3:DeleteObject. What happens when the principal tries to delete?

Quiz

You set up a role for a third-party SaaS to assume in your account, trusting their AWS account ID. Why add an sts:ExternalId condition to the trust policy?

Recall before you leave
  1. 01
    State the full IAM evaluation rule, the role each policy type plays, and how effective permissions are computed.
  2. 02
    Explain KMS envelope encryption end to end and the one key-policy mistake that orphans a key.
Recap

IAM permissions are decided by a fixed evaluation order, not by which policy looks most specific. Every request begins as an implicit deny; an explicit allow in any applicable policy can grant it; and an explicit deny anywhere — identity policy, resource policy, permissions boundary, or SCP — always wins and cannot be overridden. Now when you see an “Access Denied” error and the identity policy clearly grants the action, your first instinct should be to look for an explicit deny or a boundary/SCP that caps the ceiling — not to add more allows. Effective permissions are the union of allows, intersected with every applicable boundary and SCP, minus any explicit deny: boundaries and SCPs cap the ceiling, the union of allows fills in below it, and the deny is the kill switch. Conditions (aws:SourceIp, aws:PrincipalOrgID, aws:SecureTransport, tag matches) turn broad statements into real least privilege, and roles issuing short-lived STS credentials should replace long-lived access keys everywhere — the leaked, unrotated, wildcard-scoped key in the war story failed all three guardrails at once. Cross-account access is a role whose trust policy names another account, and the confused deputy is closed with an ExternalId (or aws:SourceArn and aws:SourceAccount for service principals). For data, KMS envelope encryption lets one central key that never leaves its HSM wrap per-object data keys, so encryption scales without ever shipping bulk data to KMS — but a key policy that omits your account root orphans the key forever. Finally, secrets belong in Secrets Manager, where a Lambda rotation function rotates database credentials with zero downtime, or in the cheaper Parameter Store SecureString for static, low-churn config. Explicit deny wins, boundaries and SCPs cap, conditions scope, roles beat keys, ExternalId stops the deputy, KMS envelopes the data, and Secrets Manager keeps it fresh.

Practice

Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.

recallapplystretch0 of 5 done

Something unclear?

Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.

Apply this

Put this lesson to work on a real build.

shortcuts expand
search
K
prev piece
k
next piece
j
cycle tier
t
this menu
?
sources4
expand
  1. 01
  2. 02
  3. 03
  4. 04

Trademarks belong to their respective owners. Editorial reference only.