Infrastructure as Code: the plan, the state file, and the drift

Two engineers ship a hotfix on Friday. Both run terraform apply from their laptops against the same project, ninety seconds apart. There is no remote lock — state lives in an S3 bucket but nobody wired up the lock table. The second apply reads a state file the first has not finished writing, computes a plan against a half-written reality, and recreates a load balancer that already existed. Production drops connections for four minutes, and the terraform.tfstate in the bucket now disagrees with what AWS actually has. The outage was not a bad config. It was two writers and no lock.

Declarative: you describe the destination, not the route

Clicking through a cloud console is imperative — you perform steps, and the result lives only in the provider and in your memory. Infrastructure as Code flips this: you write the desired end state in version-controlled files (HCL for Terraform/OpenTofu, real TypeScript/Go/Python for Pulumi), and the tool figures out the steps to get there. You do not say “create a VPC, then a subnet, then a NAT gateway.” You declare that those resources exist with these properties, and the tool computes the dependency graph and the order.

This is the whole payoff. Because the desired state is text in git, your environments become reproducible (spin up an identical staging from the same module), reviewable (infra changes go through pull requests like any code), and auditable (the diff and the blame history say who changed what and when). The console gives you none of that — a change there leaves no review, no diff, no history beyond a thin audit log.

The price of declarative is idempotency: running apply twice must converge to the same result, not stack up duplicates. The tool guarantees this only because it remembers what it already built — which is exactly what the state file is for.

The plan/apply cycle is a diff engine

The core loop is two commands. terraform plan reads three things — your config (desired state), the recorded state file (what the tool last built), and reality (it refreshes by querying the provider) — and prints the diff: what it will create, change, or destroy. Nothing happens yet. terraform apply executes that diff and writes the new reality back into the state file.

Read that again: plan and apply both run a refresh first, reconciling the state file against the live provider before computing anything. That refresh step is where drift surfaces, and it is why the state file is not optional bookkeeping — it is the map from your config’s symbolic names (aws_lb.web) to real provider IDs (arn:aws:elasticloadbalancing:...). Lose that map and the tool no longer knows which real resource your code refers to.

The state file is the heart and the hazard

Everything good about IaC routes through the state file, and so does everything dangerous. Three properties make a senior treat it with care.

First, it can hold secrets in plaintext. Terraform, OpenTofu, and Pulumi all serialize resource attributes into state — so if a database password, an API key, or a generated certificate is an output or an attribute, it sits unencrypted in the file by default. Anyone with read access to that bucket has your secrets. The mitigation is to never route secrets through state as outputs: write them straight to a secrets manager during apply and have apps fetch them at runtime. OpenTofu added built-in state encryption to harden this; Pulumi encrypts secret values per-stack as a first-class feature.

Second, it must live in a remote backend, not on a laptop. Local state means one person owns reality and the team can’t collaborate. The standard is a remote backend — S3, GCS, an OpenTofu/Terraform cloud backend — with versioning on, so a corrupted or truncated state can be rolled back to the last good version.

Third, it must be locked. This is the one that ends incidents.

Locking: why concurrent applies corrupt state

A write to the state file is not atomic across a team. If two applies run at once, both read the old state, both compute plans against it, and both write back — the second clobbers the first, and now the file describes neither reality. The fix is a lock: before any write operation, the backend acquires an exclusive lock (for S3, the native lockfile via use_lockfile = true is now the default path, with DynamoDB still valid as the legacy mechanism), and any second apply waits or fails fast with Error acquiring the state lock instead of racing.

The senior knows the sharp edges here. terraform apply -lock=false disables this and is how you reproduce the Hook outage on purpose. terraform force-unlock exists for stale locks left by a crashed run — but running it while another apply is actively writing leaves state half-updated and corrupt. The discipline in CI: a concurrency group so jobs never overlap, a -lock-timeout (say 10m) so legitimate in-flight runs are waited on rather than failed, and a plan after any forced unlock to verify state is consistent before the next apply.

Drift: when reality wanders off

Drift is when the real world diverges from the state file — almost always because someone changed infra out-of-band (a console click, an emergency aws cli patch, a different tool). The next plan refreshes, sees the difference, and reports it. Now you face a senior judgment call: is the manual change correct (then update your config to match, so the next apply doesn’t revert it) or is it unwanted (then let apply restore the declared state)?

The trap is the silent revert. Someone hand-bumps a security group rule in an incident at 2am; nobody updates the code; a routine apply on Tuesday quietly removes the rule because it isn’t in the desired state. IaC will always fight drift back to the declaration — that is the feature, and the foot-gun. Detect it deliberately with terraform plan -refresh-only (safer than the bare refresh, which overwrites state without showing you), ideally on a schedule, so drift is reviewed before an apply silently resolves it.

The deeper cure is immutable infrastructure: stop hand-mutating servers and instead replace them — bake a new image, roll it out, destroy the old. When nothing is mutated in place, there is far less surface for drift, and rollback is “deploy the previous image” instead of “remember every manual tweak.”