ECS internals: task lifecycle, awsvpc networking, capacity providers

Task definition, task, service — and the lifecycle the scheduler runs

Three nouns get used interchangeably and they are not the same thing. A task definition is an immutable, versioned blueprint: which containers, their image, CPU/memory, the two IAM roles, the network mode, port mappings, and log config. Registering it produces a numbered revision (myapp:7) you never edit — you register a new revision. A task is one running instantiation of a revision: actual containers on actual capacity with a real network interface. A service is the controller that keeps N tasks of a given definition running, registers them with a load balancer, and drives autoscaling. The service runs a continuous reconciliation loop: compare desired count to running count, and if they differ, launch or stop tasks until they match — the same converge-to-desired-state model as a Kubernetes controller.

A task does not blink into existence. It walks a lifecycle, and knowing the states tells you where a stuck task is stuck:

PROVISIONING → PENDING → ACTIVATING → RUNNING
            → DEACTIVATING → STOPPING → DEPROVISIONING → STOPPED

In PROVISIONING, ECS sets up prerequisites the task needs before launch — for awsvpc tasks, this is where the ENI is allocated and attached. PENDING is a wait-for-capacity state: the scheduler is waiting on resources or the agent. ACTIVATING is where the agent pulls images, creates containers, wires up networking, and registers load-balancer targets. RUNNING is steady state. On the way down, DEACTIVATING drains the load balancer, STOPPING sends SIGTERM (then SIGKILL after the stop timeout), DEPROVISIONING detaches and deletes the ENI, and STOPPED is terminal. The failure-mode payoff: a task wedged in PROVISIONING is a resource problem (no ENI, no IP), a task cycling through RUNNING → STOPPED repeatedly is a runtime problem (crash, failed health check). The state names the layer.

{
  "family": "checkout",
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "memory": "2048",
  "executionRoleArn": "arn:aws:iam::111122223333:role/ecsTaskExecutionRole",
  "taskRoleArn":      "arn:aws:iam::111122223333:role/checkoutTaskRole",
  "containerDefinitions": [
    {
      "name": "checkout",
      "image": "111122223333.dkr.ecr.eu-central-1.amazonaws.com/checkout:7",
      "secrets": [
        { "name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:eu-central-1:111122223333:secret:checkout/db-AbCdEf" }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/checkout",
          "awslogs-region": "eu-central-1",
          "awslogs-stream-prefix": "checkout"
        }
      }
    }
  ]
}

Note the two roles, because confusing them is the single most common ECS bug. The executionRoleArn is assumed by the ECS agent — AWS infrastructure — at launch, before your code runs, to pull the image from ECR, write to CloudWatch Logs, and decrypt the secrets values from Secrets Manager / SSM. The taskRoleArn is assumed inside the container by your application code for its own AWS calls (read S3, query DynamoDB). They are not interchangeable: a missing execution-role permission means the task never starts (CannotPullContainerError, or ResourceInitializationError decrypting a secret); a missing task-role permission means the app runs fine but gets AccessDenied at the API call.

awsvpc networking: one ENI per task, and ENIs are finite

Why does networking mode matter enough to dedicate a whole section to it? Because awsvpc gives you a clean VPC citizen per task — its own security group, its own IP, its own flow logs — but that elegance has a hard countable cost that causes outages at exactly the worst time: when you scale.

In awsvpc mode — mandatory for Fargate, the modern default for EC2 — each task gets its own elastic network interface (ENI) with a private IP in your subnet. That’s the feature: a task is a first-class VPC citizen with its own security group, its own IP, and direct flow logs, instead of sharing the host’s interface and juggling ports (bridge/host mode). The cost is that an ENI and an IP are real, countable resources, and you can run out of both.

Two limits bite. First, subnet IPs: a /24 subnet has 256 addresses, AWS reserves 5, so 251 usable — meaning at most ~251 awsvpc tasks across everything sharing that subnet, and that’s where the Hook’s scale-out died (RESOURCE:ENI is also raised on IP exhaustion). Second, on EC2 capacity, ENIs-per-instance is capped by instance type: a c5.large allows 3 ENIs total, the primary counts as one, so you can run only 2 awsvpc tasks on it — bin-packing dense tiny tasks hits the ENI wall long before CPU or memory. ENI trunking (awsvpcTrunking account setting) attaches a shared trunk interface so the same c5.large jumps to an effective 10 tasks, but only on instances launched after you enable it. Fargate sidesteps the per-instance ENI limit entirely — every task gets a dedicated ENI no matter how many you launch — but each one still consumes a subnet IP, so Fargate does not save you from subnet exhaustion. The senior reflex: size subnets for peak task count plus deploy headroom (rolling deploys briefly run new + old), and use a /22 or /21 for anything that scales, never a /26.

# Why is the scale-out stuck? Read the service events — they name the cause.
aws ecs describe-services --cluster prod --services checkout \
  --query "services[0].events[0:5].message"
# => "... unable to place a task because no container instance met all of
#     its requirements ... has insufficient ENI capacity. RESOURCE:ENI"

# How many free IPs does the task subnet actually have right now?
aws ec2 describe-subnets --subnet-ids subnet-aaa \
  --query "Subnets[0].{Cidr:CidrBlock,Free:AvailableIpAddressCount}"
# => { "Cidr": "10.0.1.0/24", "Free": 0 }   <-- there's your outage

Capacity providers: Fargate vs EC2, and how to choose

A capacity provider answers “where do tasks actually run?” There are two families. Fargate is serverless: no hosts to patch, scale, or bin-pack; you pay per-task vCPU/memory by the second (1-minute minimum); it’s the fastest to operate and the right default. The price is a per-vCPU premium over raw EC2, and less control — no daemon containers, no GPUs, limited instance-type and kernel-tuning options. EC2 capacity providers run your own instances in an Auto Scaling group fronted by the provider, which manages managed scaling (grows/shrinks the ASG to fit pending tasks) and managed termination protection (won’t scale in an instance still running tasks). EC2 is cheaper at steady high utilization, supports Spot for ~70% savings on interruptible work, GPUs, and daemonsets — but you own patching, AMIs, the ENI-density math above, and cluster bin-packing. Fargate Spot offers the same interruption discount without managing hosts, for jobs that tolerate a 2-minute termination notice.

{
  "capacityProviders": ["FARGATE", "FARGATE_SPOT"],
  "defaultCapacityProviderStrategy": [
    { "capacityProvider": "FARGATE",      "base": 4, "weight": 1 },
    { "capacityProvider": "FARGATE_SPOT", "base": 0, "weight": 4 }
  ]
}

That strategy reads: keep a guaranteed base of 4 tasks on on-demand Fargate (your floor, never interrupted), then split everything above the base 1:4 — for every 1 on-demand task, run 4 on Spot. It’s the canonical “stay-up baseline on-demand, burst cheap on Spot” pattern, and the same shape works for an EC2 + EC2-Spot mix.

Why a task won’t start: reading stoppedReason

When a task dies, ECS records why on the stopped task: a coarse stopCode and a human stoppedReason. This is the senior debugging entry point — you read these before guessing.

aws ecs describe-tasks --cluster prod --tasks <task-arn> \
  --query "tasks[0].{Code:stopCode,Reason:stoppedReason,
                     Containers:containers[].{N:name,Reason:reason,Exit:exitCode}}"

The checklist, keyed to what you’ll see:

• CannotPullContainerError — image pull failed. Causes: missing/insufficient execution role, a private subnet with no NAT/VPC endpoints to reach ECR (very common), wrong image tag, or an arm64/amd64 mismatch. Task dies in PENDING/ACTIVATING.
• RESOURCE:ENI / RESOURCE:MEMORY / RESOURCE:CPU (stopCode: TaskFailedToStart / placement event) — the cluster has no room: subnet IPs exhausted (ENI), or no EC2 instance with enough CPU/memory. Stuck in PROVISIONING/PENDING.
• ResourceInitializationError — usually decrypting a secrets value: the execution role lacks secretsmanager:GetSecretValue / ssm:GetParameters (or kms:Decrypt on the key).
• Essential container ... exited with a non-zero exitCode — your app crashed, or it failed the container/ALB health check and ECS killed it. In a service this becomes a deploy loop: new tasks fail the check, ECS won’t promote them, old tasks stay. The deployment circuit breaker (enable: true, rollback: true) detects the loop and rolls back to the last good revision instead of churning forever.

Numbers to hold: a Fargate task is typically RUNNING in tens of seconds (ENI attach + image pull); image pull alone is seconds to minutes by image size, which is why slim images and pull-through cache matter. The stop timeout (SIGTERM → SIGKILL) defaults to 30 seconds. None of this is guesswork — stoppedReason points at the layer; you fix that layer.