Autoscaling and load balancing: ALB, NLB, and target tracking

ALB vs NLB: layer 7 routing or layer 4 throughput

When you reach for a load balancer, the first question isn’t “which one is faster” — it’s “does my backend need HTTP-level routing, or does it need raw TCP throughput?” The answer locks the choice before you look at any other property.

Elastic Load Balancing gives you two production load balancers, and the choice is structural, not cosmetic. An Application Load Balancer (ALB) operates at layer 7 — it terminates HTTP/HTTPS, parses each request, and routes on host, path, header, method, or query string to different target groups. That is what lets one ALB fan a single domain across many microservices: /api/* to one service, /static/* to another, host: admin.example.com to a third. Targets can be EC2 instances, IP addresses, Lambda functions, or — the common case here — ECS tasks registered automatically by the service. Because it understands HTTP, ALB also does TLS termination, sticky sessions via cookies, and per-request routing decisions.

A Network Load Balancer (NLB) operates at layer 4 — it forwards TCP and UDP flows without reading the payload. It does not parse HTTP, cannot route on path, and cannot speak to Lambda targets. What it gives you in return is ultra-low latency (no request parsing), the ability to handle millions of flows per second, a static IP per Availability Zone (and an Elastic IP option), native PrivateLink endpoints, and — critically for some backends — it preserves the client source IP all the way to the target. The tradeoff is plain: ALB is the default for HTTP microservices that need routing, TLS, and rich health checks; NLB is for raw TCP/UDP, extreme throughput, a fixed IP that firewalls and allowlists can pin, or anything that must see the real client IP.

# ALB: layer 7, routes by path to two ECS target groups.
resource "aws_lb_listener_rule" "api" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 10
  action { type = "forward"; target_group_arn = aws_lb_target_group.api.arn }
  condition { path_pattern { values = ["/api/*"] } }
}

# NLB: layer 4, raw TCP passthrough on 5432 to a Postgres proxy fleet.
resource "aws_lb_listener" "pg" {
  load_balancer_arn = aws_lb.nlb.arn
  port              = 5432
  protocol          = "TCP"
  default_action { type = "forward"; target_group_arn = aws_lb_target_group.pgproxy.arn }
}

The thing that actually decides whether a target receives traffic is the target group health check: a path (for ALB, e.g. GET /healthz), an interval, a timeout, and healthy/unhealthy thresholds (how many consecutive successes mark a target healthy, how many failures mark it unhealthy). This is a live filter — a target stays out of rotation until it passes, and is pulled the instant it fails the threshold. Tune it wrong both ways: a too-aggressive check (short interval, low unhealthy threshold, tight timeout) flaps healthy targets out of rotation on a single slow GC pause or a brief dependency blip, shrinking your fleet exactly when load is high; a too-lax check (long interval, high threshold) keeps routing to a dead or hung target for tens of seconds, so users hit 5xx until the check finally notices.

Target groups and the deregistration delay

When a target leaves — scale-in removed a task, or a deploy is replacing one — the load balancer does not just yank it. It runs connection draining, governed by the target group’s deregistration delay (attribute deregistration_delay.timeout_seconds, default 300 seconds). On deregistration the LB immediately stops sending the target new requests but lets in-flight requests finish, for up to the delay; only then does the target fully detach. This is the difference between a clean rolling deploy and a wall of errors.

resource "aws_lb_target_group" "api" {
  name        = "api-tg"
  port        = 8080
  protocol    = "HTTP"
  target_type = "ip"            # ECS awsvpc tasks register by IP

  health_check {
    path                = "/healthz"
    interval            = 15     # probe every 15s
    timeout             = 5
    healthy_threshold   = 2      # 2 in a row → in rotation
    unhealthy_threshold = 3      # 3 in a row → pulled
    matcher             = "200"
  }

  # Drain in-flight requests on scale-in / deploy. Match this to your
  # real p99 request duration plus margin — NOT the 300s default blindly.
  deregistration_delay = 30

  stickiness { type = "lb_cookie"; enabled = false }
}

The failure mode is a two-sided trap. Set the deregistration delay too short and every scale-in or deploy severs in-flight requests mid-flight: a 12-second report export or a long-poll gets killed at second 5, and you ship a 5xx to a real user on every deployment — a self-inflicted error spike that correlates suspiciously with your release times. Set it too long (the 300s default for an app whose requests finish in 200ms) and every rolling deploy crawls, because each task you remove sits draining for five idle minutes before the deploy can declare it gone — slow rollouts, slow rollbacks, and a much larger window where old and new code run side by side. The right value is your real p99 request duration plus a margin, not the default.

The other target-group lever is cross-zone load balancing. With it on (the default for ALB; off by default for NLB), each LB node can send to targets in any AZ, so traffic spreads evenly even when AZs hold uneven target counts — but cross-AZ traffic on NLB is billed as inter-AZ data transfer. With it off, each LB node only hits same-AZ targets, which is free but skews load if one AZ has 2 targets and another has 8: the 2 absorb the same share of that zone’s traffic as the 8 do of theirs, so the small zone runs hot.

Autoscaling: the control loop and its built-in lag

Both ECS service autoscaling (via Application Auto Scaling) and EC2 Auto Scaling Groups drive capacity from CloudWatch metrics, and the policy types are the same three. Target tracking is the senior default: you name a metric and a target value — CPU at 50%, or ALBRequestCountPerTarget at 1000 — and AWS manages the alarms and adds/removes capacity to hold the metric near that target, like a thermostat. Step scaling adds a tiered response (breach by a little → +1, breach by a lot → +4) when you need explicit control. Scheduled scaling sets capacity by clock for known patterns — scale to 20 tasks at 08:55 because the workday starts at 09:00.

{
  "PolicyName": "track-requests-per-target",
  "PolicyType": "TargetTrackingScaling",
  "TargetTrackingScalingPolicyConfiguration": {
    "TargetValue": 1000.0,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/prod-alb/abc123/targetgroup/api-tg/def456"
    },
    "ScaleInCooldown": 120,
    "ScaleOutCooldown": 30
  }
}

The control loop is the whole story, and every stage costs wall-clock time: metric emitted → CloudWatch aggregates it (~1 minute) → alarm needs N datapoints over M minutes to breach → scaling action requested → new task/instance must launch, warm up, and pass the health-check thresholds before the LB routes to it. Nothing serves traffic until that last step. Cooldowns gate the loop further: too-short cooldowns make it flap — scale out, the metric drops, scale in, the metric rises, scale out again — thrashing capacity and paying launch costs on every cycle; too-long cooldowns leave you stuck under-provisioned through a sustained climb. Note the asymmetry in the policy above: scale out fast (short cooldown — being late costs users) and scale in slow (long cooldown — being hasty costs a re-spin).

The scale-out lag: why a spike still returns 5xx

Here is the war story made general. Autoscaling is reactive — it can only respond after a metric has moved, been aggregated, and breached an alarm — and a sudden 10× spike does not grant that grace period. Add the stages: metric aggregation ~1 minute, alarm evaluation across its datapoints ~1–3 minutes, then launch and warmup. A Fargate task starts in tens of seconds; an EC2 instance launch plus OS boot, agent registration, image pull, and app warmup is minutes. Sum it and your new capacity is often 3–5 minutes behind the spike — an eternity at 10× load, during which the current fleet alone eats the surge and returns 5xx and timeouts. The fleet was sized for steady state; the spike was not.

The senior mitigations all attack a different stage of that loop. Target tracking with headroom — set the target to CPU 50%, not 80% — so steady-state already runs with spare capacity to absorb the first wave while scale-out catches up. Scale on request count, not CPU, because it is the leading signal (see the Inset). Scheduled scaling for known peaks (the flash sale, the 09:00 login storm): pre-warm capacity by the clock so you never enter the spike cold. Predictive scaling (EC2 ASG) learns the daily/weekly cycle from history and provisions ahead of the forecast curve. Warm pools (EC2) keep pre-initialized, stopped instances that resume in seconds instead of booting from scratch. Provisioned capacity / provisioned concurrency keeps a baseline always hot. And when capacity genuinely cannot arrive in time, shed or queue: return 503 with Retry-After or push work to SQS so the spike is buffered instead of dropped. None of these make autoscaling instant — they shrink or hide the lag so the spike meets capacity instead of a wall of errors.