IaC pipelines: plan-on-PR, policy-as-code, and drift

The plan/apply pipeline: the diff is the review artifact

The previous lessons taught you to write IaC — CloudFormation and CDK, Terraform with a locked remote backend, state operations, modules across accounts. This one is about running it safely, and the shape that makes it safe is GitOps for infrastructure: every change flows through a pull request, and the pipeline — not a laptop — is the thing that talks to AWS.

The mechanism splits the workflow across two pipeline triggers. On a PR, CI runs the read-only half: terraform plan (or cdk diff, or a CloudFormation change set), then posts the resulting diff as a PR comment. On merge to the default branch, CI runs the write half: terraform apply. The crucial idea is that the plan is the review artifact. A reviewer approving the PR is not approving the HCL in the abstract — they are approving the exact change set the plan printed: these three resources update in place, this one is replaced (destroyed and recreated), nothing else. Code review of .tf files alone is necessary but not sufficient, because the same code produces a different plan depending on current state. The plan closes that gap by showing the literal delta that will hit the account.

# .github/workflows/infra.yml — plan on PR, apply on merge
name: infra
on:
  pull_request:        # PR: plan only, post the diff for review
  push:
    branches: [main]   # merge: apply

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform plan -no-color -out=tfplan
      - run: terraform show -no-color tfplan > plan.txt
      # …then a step posts plan.txt as a PR comment (the review artifact)

  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: prod   # gate: requires a human approval before this job runs
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform apply -auto-approve

The tradeoff lives in that environment: prod line. A required manual approval on apply is safe — a human re-reads the plan and clicks before prod mutates — but slow, and it puts a person in the loop for every change. Auto-applying on merge is fast and fully GitOps-pure, but a bad merge ships straight to production with nobody re-reading the plan. The failure mode of pure auto-apply is the 3am merge that destroys a database because the plan showed a -/+ replace nobody scrutinized. The senior default is asymmetric: plan-on-PR always, and apply gated by environment — auto-apply for dev (cheap to break, fast feedback), human approval for prod. You buy speed where mistakes are cheap and a gate where they are expensive.

CI to AWS without long-lived keys: OIDC federation

The pipeline now talks to AWS on every merge, which means it needs credentials — and the wrong way to give them is the way most teams start: paste a long-lived IAM user’s AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY into CI secrets. That key is valid until someone rotates it, which is often never, and it sits in your CI provider waiting to leak: into a build log via an env dump, through a compromised third-party action with access to the secret, or out of a misconfigured fork PR. When a static admin key leaks, the blast radius is the entire account, immediately and indefinitely.

The mechanism that removes the standing secret is OIDC federation. GitHub Actions (and GitLab CI) can mint a short-lived OIDC token — a signed JWT describing the workflow run — and exchange it with AWS STS via AssumeRoleWithWebIdentity for temporary credentials. You register GitHub’s OIDC provider once in IAM, then attach a trust policy to a role that says “only mint credentials for a token whose claims match this repo on this branch.” No secret is stored anywhere; the credentials AWS hands back expire in about an hour (the role’s max session, tunable), so a token captured from a log is worthless minutes later.

  apply:
    runs-on: ubuntu-latest
    environment: prod
    permissions:
      id-token: write     # let the job request an OIDC token
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/ci-terraform-prod
          aws-region: eu-central-1
          # no AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY anywhere

The whole security boundary is the trust policy’s condition. Scope the sub claim to a specific repo and ref, so a fork, a feature branch, or another org’s workflow cannot assume your prod role:

{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/infra:ref:refs/heads/main"
    }
  }
}

The tradeoff is mild — OIDC needs a one-time IAM setup (register the provider, write the trust policy) and disciplined sub conditions — against an enormous payoff: no rotation, no standing secret, and a credential that is scoped, short-lived, and auditable. The classic failure mode is a trust policy with a sub like repo:acme/infra:* (any branch, any ref): now any branch a contributor can push, including an attacker who lands a PR branch, can assume the prod role. Pin the branch, and for stronger isolation tie the role to a GitHub environment (...:environment:prod) so the environment’s own protection rules gate the assume.

Policy-as-code: block bad infra before apply

A reviewed plan still depends on a human catching every dangerous line, and humans miss “this S3 bucket is public” at 6pm on a Friday. Policy-as-code moves that judgment into the pipeline: automated rules run against the plan (or the template) and fail the build before apply if infrastructure violates a rule. The plan you already generate for review becomes the input to a policy engine.

The tools cluster by ecosystem: OPA/Conftest and Checkov and tfsec/Trivy evaluate a Terraform plan in JSON; HashiCorp Sentinel is the paid Terraform-native option; AWS CloudFormation Guard (cfn-guard) checks CloudFormation templates and any JSON/YAML with a compact rule DSL. Together these tools share the same job: turn a “someone would have caught that” assumption into a deterministic gate that fires before prod. The rules encode the things you never want to ship — no public S3 bucket, encryption required, no 0.0.0.0/0 to port 22, mandatory cost-allocation tags — and they catch the whole class of bug, not the one instance a reviewer happened to notice.

# cfn-guard rule: no S3 bucket may be public, and encryption is required
rule s3_must_be_private when Resources.*.Type == "AWS::S3::Bucket" {
  Resources.*[ Type == "AWS::S3::Bucket" ] {
    Properties.PublicAccessBlockConfiguration exists
    Properties.PublicAccessBlockConfiguration.BlockPublicAcls == true
    Properties.PublicAccessBlockConfiguration.IgnorePublicAcls == true
    Properties.BucketEncryption exists
  }
}

# Conftest/OPA equivalent over a `terraform show -json` plan
package main
deny[msg] {
  r := input.resource_changes[_]
  r.type == "aws_security_group_rule"
  r.change.after.cidr_blocks[_] == "0.0.0.0/0"
  r.change.after.to_port == 22
  msg := sprintf("SSH open to the world on %s", [r.address])
}

The tradeoff is friction versus coverage. Policy gates catch bug classes pre-prod and encode tribal knowledge as enforceable rules, but they add a step that fails builds, and a noisy rule with false positives trains the team to ignore or bypass it — the worst outcome, because a routinely-overridden gate is theater. The failure mode is rolling out twenty strict rules on day one: the pipeline goes red on legitimate changes, people start adding skip annotations, and the policy layer is dead within a sprint. Roll out high-value rules first (public S3, open SSH, unencrypted volumes) in warn mode, measure the false-positive rate, then flip them to enforce once they’re clean. A few enforced rules people trust beat fifty they route around.

Drift and concurrency: the two ways the apply itself bites

The Hook is the canonical IaC incident, and it has a name: drift — reality diverging from state because something changed AWS outside the pipeline (a console hotfix, a script, another team). The next plan sees the divergence and wants to revert it to match the code, silently undoing the manual change; or, if the manual change conflicts with the config, the apply errors. Either way the out-of-band fix is lost or causes a surprise. The mechanism to stay ahead of it is scheduled drift detection: run terraform plan (or CloudFormation’s native drift detection) on a nightly schedule, and if the plan is non-empty when no PR is open, alert the team — drift caught at 03:00 by a cron is a ticket; drift caught by a surprise revert mid-deploy is an incident. Catch it early and you decide deliberately whether to codify the change (write it into the config) or roll it back, instead of the pipeline deciding for you.

  drift:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      # detailed-exitcode: 0 = no drift, 2 = drift present, 1 = error
      - run: terraform plan -detailed-exitcode -no-color || echo "DRIFT" >> alert
      # …a non-empty plan with no open PR pages on-call
on:
  schedule:
    - cron: "0 3 * * *"   # nightly drift scan

The second hazard is concurrency. Two pipelines applying the same state at once race: both read the same starting state, both write, the second clobbers the first, and the state file is now inconsistent with reality — orphaned resources Terraform no longer tracks, a serial counter that no longer matches. The defense is the state locking from the Terraform lesson — the DynamoDB lock (or use_lockfile) for the S3 backend, and CloudFormation’s own server-side stack locking — plus serializing applies per state in CI (a concurrency group so two apply jobs for the same state can’t overlap). The failure mode that locking itself introduces: a CI job dies mid-apply (the runner is killed, the network drops) and leaves the lock held, so every subsequent apply blocks — Error acquiring the state lock — until someone investigates. The dangerous reflex is terraform force-unlock to “just unblock the team.” If that lock actually belongs to an apply still mutating AWS, force-unlocking lets a second apply run concurrently and corrupts the state for real. Force-unlock only after you have confirmed the holding job is truly dead; the stuck lock is annoying, the wrongful force-unlock is a corrupted prod state file.