Modules and multi-account: composition, environments, blast radius

Modules and composition: the contract and the two failure modes

A module is a reusable, parameterized unit of infrastructure — a Terraform module, a CDK construct, or a CloudFormation nested stack / StackSet. Its inputs and outputs form a contract: callers pass variables in, consume outputs, and treat the internals as a black box. The mechanism that makes this safe at scale is versioning: you publish the module to a registry (Terraform Registry, a Git tag, an S3/Artifactory artifact) and consumers reference an exact version, so a reviewed, tested building block stays frozen until someone deliberately upgrades it.

# A versioned module call — the version constraint is the whole point.
module "vpc" {
  source  = "app.terraform.io/acme/vpc/aws"
  version = "4.2.1"          # pinned. NOT "~> 4.0" on a hot path, NEVER a bare branch ref.

  cidr_block          = var.cidr_block
  availability_zones  = var.azs
  enable_nat_gateway  = var.env == "prod"   # input, not a forked copy
}

# Consumers wire on the module's OUTPUT contract, not its internals.
module "app" {
  source     = "app.terraform.io/acme/ecs-service/aws"
  version    = "2.7.0"
  subnet_ids = module.vpc.private_subnet_ids   # the contract
}

The tradeoff is a genuine sweet spot with a cliff on each side. Under-abstraction — every team copy-pastes its own VPC HCL — means the same fix has to be made in a dozen places, and they drift: dev’s VPC slowly diverges from prod’s until “works in stage” stops predicting prod. Over-abstraction is the opposite trap: a god-module with 60 input variables and a dozen feature flags that nobody fully understands, where a one-line change ripples through every consumer and the abstraction obscures more than it hides. The failure mode of the god-module is that it becomes harder to reason about than the duplication it replaced — and, as the hook showed, a single edit now has reach across every stack that calls it.

The senior rule: build a module for a genuinely repeated pattern with a stable interface, not for everything. If you’ve written the same VPC three times, modularize it. If you’re abstracting a thing used once, you’ve added indirection with no payoff. A good module is small, has a narrow input surface, and changes rarely; a 60-variable god-module is a smell, not a win.

Environment promotion: same code, different inputs

The discipline that makes prod actually match stage is this: the same module version runs in every environment, and environments differ only by inputs — tfvars files, CloudFormation parameter files, or CDK context — never by forked code. The moment dev and prod are different code rather than the same code with different variables, “it passed in stage” stops being evidence about prod.

# envs/prod/terraform.tfvars  — same module, prod inputs
env        = "prod"
cidr_block = "10.0.0.0/16"
azs        = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
instance_count = 6

# envs/dev/terraform.tfvars   — same module, dev inputs
env        = "dev"
cidr_block = "10.9.0.0/16"
azs        = ["eu-central-1a"]
instance_count = 1

How you separate the environments has a direct blast-radius consequence — and this is where lesson 03’s state model pays off. Workspaces keep one configuration and switch the state slice (terraform workspace select prod); cheap, but one wrong select and you apply prod’s plan against dev’s intent, and all envs share one backend. Directory-per-environment gives each env its own root and its own state file, so a corrupt or locked dev state can never touch prod’s state. Separate state per environment is the senior default precisely for that isolation. Promotion is then a deliberate act: you apply module version 4.2.1 to dev, let it bake and pass checks, then apply that same version to stage, then prod — never letting prod silently float to a version stage hasn’t seen.

The failure mode of getting this wrong is subtle: a team uses workspaces “to keep it simple,” someone runs apply in the prod workspace believing they’re in dev, and there is no second boundary to catch it. Directory- and account-separated environments turn that fat-finger into a wrong-directory error you notice, not a prod outage you cause.

Multi-account: the real blast-radius boundary

State isolation protects you from IaC mechanics; it does not stop a runaway resource or a bad IAM policy in one environment from reaching another inside the same AWS account. The strongest boundary AWS gives you is the account itself. Under AWS Organizations, you run an account-per-environment (often also per-team or per-workload): a blast in dev cannot touch prod because IAM principals, service quotas, and billing are all scoped per account. A fat-fingered iam:* policy in dev grants nothing in prod; a runaway Lambda recursion that bills $40k stays inside dev’s bill, not prod’s.

Service Control Policies (SCPs) are the org-wide guardrail: they set the maximum permissions any principal in an account can ever have — even the account root and admins. An SCP that denies an action class applies to everything below it, regardless of IAM grants:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyPublicS3OrgWide",
      "Effect": "Deny",
      "Action": "s3:PutBucketPublicAccessBlock",
      "Resource": "*",
      "Condition": {
        "Bool": { "s3:PublicAccessBlockEnabled": "false" }
      }
    }
  ]
}

Control Tower (a landing zone) automates this baseline — org structure, a security/log-archive account, and SCP guardrails — so new accounts come up pre-fenced. To deploy into many accounts you use CloudFormation StackSets (one template, fanned out to a target set of accounts and regions) or, in Terraform, provider aliases with assume-role — one config that assumes a role per account and applies there:

provider "aws" {
  alias  = "prod"
  region = "eu-central-1"
  assume_role { role_arn = "arn:aws:iam::111111111111:role/terraform-deploy" }
}
provider "aws" {
  alias  = "dev"
  region = "eu-central-1"
  assume_role { role_arn = "arn:aws:iam::222222222222:role/terraform-deploy" }
}

Cross-stack wiring and the shared-module blast

Real estates aren’t one stack — they’re many, and they reference each other. The network account exports a VPC ID; the app stack imports it. The mechanisms: Terraform’s terraform_remote_state data source reads another state’s outputs; SSM Parameter Store holds shared values one stack writes and others read; CloudFormation exports (Fn::ImportValue) link stacks by name.

data "terraform_remote_state" "network" {
  backend = "s3"
  config  = { bucket = "acme-tf-state", key = "prod/network/terraform.tfstate", region = "eu-central-1" }
}

resource "aws_lb" "app" {
  subnets = data.terraform_remote_state.network.outputs.public_subnet_ids
}

This wiring buys composition but creates ordering and coupling: the producer must apply before the consumer, and you cannot delete an exported value while another stack still imports it — CloudFormation rejects the delete, and Terraform’s consumer plan breaks. A “harmless” rename of an output becomes a cross-stack incident.

The sharpest edge is the shared-module blast from the hook. Bumping a widely-used module’s version — or, worse, referencing a moving target like a branch or latest — silently changes every consumer on their next apply. With the module backing 40 stacks, an unpinned source means the next routine apply in any of them pulls the new code and proposes a change nobody requested. The fix is mechanical and non-negotiable: pin module versions, bump them in a deliberate PR, and roll the new version through dev → stage → prod like any other change. The reach of a single edit is exactly the count of consumers; pinning makes that reach opt-in instead of automatic. The same logic scales SCPs: one statement at the org root can block an entire action class across N accounts at once — enormous leverage for a guardrail, and an enormous outage if it’s wrong, so SCPs roll out like code too.