Autoscaling with Actions Runner Controller: ephemeral pods that scale to the queue
Actions Runner Controller (ARC) runs self-hosted runners as ephemeral Kubernetes pods. A runner scale set scales pods up to maxRunners, each servicing one job, giving hosted-like isolation on your cluster with capacity that tracks the job queue instead of a fixed pool.
The pool was eight static self-hosted runners on EC2, always on, and at 9 a.m. on a Monday it became a bottleneck: forty engineers pushed before standup, the queue grew to 60 jobs, and the median wait time hit eleven minutes for a test suite that ran in four. The runners were never idle and never enough. The platform team’s first instinct was to add static capacity — but eight machines billed 24/7 were already burning money overnight when the queue was empty, and doubling them would double the idle waste to fix a two-hour-a-day spike. They moved to Actions Runner Controller instead: runners became ephemeral Kubernetes pods that scaled from zero to forty when the queue filled and back to zero when it drained. Monday’s wait time dropped to under a minute, and the overnight bill dropped to nothing because no pods ran when no jobs were queued. The fixed pool had been paying for peak capacity around the clock to serve a spike that lasted two hours.
What ARC is
If you have ever asked “why am I paying for idle runners overnight to handle a two-hour morning spike?”, ARC is the answer — but only if you understand what actually drives its scaling decisions.
Actions Runner Controller is a Kubernetes operator that manages self-hosted runners as pods. You install the controller and define a runner scale set — a Helm release bound to a repository, organization, or enterprise with a registration token. The scale set’s listener holds a long poll to GitHub’s Actions service; when jobs are assigned to that scale set’s runner group, the listener learns how many are queued and the controller creates that many ephemeral runner pods. Each pod registers, runs exactly one job, then terminates and is replaced. There is no persistent runner accumulating state between jobs — the same one-job-one-runner isolation that makes hosted runners safe, now on your own cluster.
The scaling is bounded by two values you set: minRunners (a warm floor of idle pods to absorb the cold-start latency of pod scheduling and image pull) and maxRunners (the ceiling that caps both your blast radius and your cloud bill). Between them, pod count tracks the queue: jobs queued drives pods up toward maxRunners, jobs draining lets pods finish and not be replaced, falling back toward minRunners (which can be zero — scale-to-zero when idle).
In an ARC runner scale set, how many jobs does a single ephemeral runner pod handle before it is replaced, and why does that matter?
Cold start, warm floor, and the queue-wait tradeoff
The cost of ephemerality is cold start: a freshly created pod must be scheduled onto a node (which may itself need the cluster autoscaler to add a node — tens of seconds to minutes), pull the runner image, register with GitHub, and only then pick up the job. For a four-minute test suite, a 90-second cold start is a 38% overhead on the first job of a spike. minRunners is the lever: keep a few pods warm and idle so the front of a burst lands on a ready runner while the controller scales the rest behind it. The tradeoff is direct — warm pods cost money while idle, so minRunners is a dial between queue-wait latency (set it low, pay less, wait more at the start of a burst) and responsiveness (set it higher, pay for idle warmth, start faster). Right-size it to your actual spike shape, not to peak.
▸lesson.inset.connect
ARC layers on the cluster autoscaler, and the two cold starts compound. When the queue spikes past what current nodes can hold, ARC creates pending pods that no node can schedule; the Kubernetes cluster autoscaler sees the unschedulable pods and provisions new nodes. So a cold burst can pay two latencies: node provisioning, then pod startup. Teams that need fast spike response keep both a warm minRunners floor and a small headroom of spare node capacity, accepting steady idle cost to avoid the stacked cold-start delay — the same fixed-cost-versus-latency tradeoff as the earlier static pool, now tuned with two knobs instead of one.
Sizing maxRunners and the runner image
maxRunners is your safety ceiling. Set it from two constraints: your GitHub Actions concurrency limit (a plan-level cap on concurrent jobs — exceeding it just queues, so a maxRunners above it wastes nodes) and your cluster capacity and budget (each pod consumes CPU/memory requests; maxRunners times the per-pod request must fit your nodes, and uncapped scaling can run up a cloud bill or starve other workloads on a shared cluster). The pod image matters too: a lean custom runner image with your toolchain pre-baked cuts the per-pod cold start dramatically versus pulling a fat image and installing tools at job start — the same logic as a well-built CI base image, applied to the runner pod itself.
A team sets minRunners: 0 to minimize cost and reports that the first job after an idle period waits about two minutes before it even starts. What is the most likely cause and the direct fix?
- 01Describe the ARC scaling loop and the roles of minRunners and maxRunners.
- 02Why does scale-to-zero (minRunners: 0) introduce first-job latency, and what compounds it?
Actions Runner Controller is a Kubernetes operator that replaces a fixed pool of always-on self-hosted runners with ephemeral pods that scale to the job queue. You define a runner scale set bound to a repo, org, or enterprise; its listener long-polls GitHub’s Actions service for queued jobs, and the controller reconciles the pod count to match. Each pod registers, runs exactly one job, and terminates — the same one-job-one-runner isolation that makes hosted runners safe, brought onto your own cluster. Scaling is bounded by minRunners, a warm floor of idle pods that lowers first-job latency at the cost of paying for idle warmth, and maxRunners, a hard ceiling that caps blast radius and cloud spend. The price of ephemerality is cold start: a fresh pod must be scheduled, pull its image, and register before running, and when the cluster autoscaler must add a node first, the two latencies stack. You tune that with a non-zero warm floor, a lean pre-baked runner image, and a little spare node headroom — each trading steady idle cost for faster spikes. Size maxRunners under your GitHub Actions concurrency limit (above it just queues) and within cluster capacity and budget, since every pod consumes CPU and memory requests and uncapped scaling can starve neighbours or run up a bill. The result is capacity that follows demand — zero pods when the queue is empty, scaling to the ceiling when it fills — instead of paying for peak capacity around the clock to serve a spike that lasts a couple of hours. Now when you see a team complaining about slow CI on Monday mornings and zero queue by noon, you know the fix: ARC with a calibrated warm floor, not more static runners.
Practice
Start at the top. Tasks go easiest → hardest: recall a fact, apply it to a case, then a senior-level stretch. Open one, attempt it, then reveal.
Something unclear?
Ask a question about this lesson. Questions are anonymous and go straight to the author to make the lesson better.