Runner infrastructure and CI cost at scale

Hosted vs self-hosted: who owns the machine

Before diving in: the question isn’t just “which runners are cheapest per minute.” When you factor in engineer-wait, the answer often flips completely — which is why this lesson starts with the cost model, not the feature comparison.

GitHub-hosted runners give you a clean, ephemeral VM per job with zero operations on your side — you pay per minute and never patch anything. That is the right default until one of three things bites: the bill at high volume, the spec ceiling (you need big machines, GPUs, or specific hardware the hosted catalog doesn’t offer), or network reach (the job must touch services inside your VPC). Self-hosted runners answer all three — your machines, cheaper at high volume, any spec, VPC access — but you inherit patching, scaling, and security.

That security clause is not a footnote. A self-hosted runner attached to a public repository is dangerous: a pull request from a stranger runs their code on your infrastructure. Keep self-hosted runners off public repos, or gate them behind required approval for first-time contributors.

# A job pinned to your self-hosted fleet by label.
jobs:
  build:
    runs-on: [self-hosted, linux, x64, ephemeral]  # labels select the fleet
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/build.sh

Ephemeral runners: a clean room per job

This is the practice that makes self-hosted safe. An ephemeral runner executes exactly one job and is then destroyed and recreated fresh. A persistent runner stays up and takes job after job — and accumulates state between them: leftover files in the workspace, a poisoned dependency cache, a background process spawned by a previous (possibly malicious) job. That residue is a contamination channel. A later job can read secrets left behind, hit a tampered binary, or inherit a corrupted cache — and nothing is reproducible because every job starts from wherever the last one left the box.

Ephemeral runners close all of that. Each job gets a clean room: deterministic starting state, no cross-job leakage, and a compromised job dies with its runner instead of lying in wait for the next one. The cost is spin-up time per job — you pay a fresh boot every time — which you mitigate with warm pools (a few idle runners pre-provisioned) and fast base images. The isolation is not obtainable any other way; the spin-up cost is.

# Register a self-hosted runner that exits after one job and is not re-used.
./config.sh --url https://github.com/acme/payments \
  --token "$REG_TOKEN" \
  --labels linux,x64 \
  --ephemeral            # <- destroy after a single job; no state survives
./run.sh

Autoscaling: track the queue, not a guess

A fixed fleet of N runners has two failure modes, and you can’t tune your way out of both with a constant. Too few runners and jobs queue — developers wait, and the real cost is engineer idle time, not an idle machine. Too many and you pay for idle runners sitting warm with nothing to do. Whatever N you pick is wrong at some point in the day, because demand is spiky: quiet at 3am, a thundering herd when everyone pushes before standup.

Autoscaling makes N follow demand. Actions Runner Controller (ARC) runs ephemeral runners as Kubernetes pods that scale with the job queue: when the queue grows, it spins up more runner pods; when it drains, it scales to zero so idle hours cost nothing. You get the best of both — capacity when there’s a herd, no bill when there isn’t — and crucially the runners are still ephemeral, so autoscaling and clean-room isolation compose instead of fighting.

# ARC: an autoscaling set of ephemeral runners (scales with queue, down to zero).
apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
spec:
  githubConfigUrl: https://github.com/acme/payments
  minRunners: 0          # scale to zero when the queue is empty
  maxRunners: 200        # ceiling so a herd can't bankrupt you
  template:
    spec:
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner:latest

The cost curve has two axes

Here is the senior reframe the war-story team missed. Ask yourself: when your team was debating the runner bill, was anyone measuring how many engineer-hours were queued up waiting? CI cost is not one number. It is the runner $ plus the engineer-time $ of everyone waiting on CI. Optimize only the bill — fewer runners, smaller machines, capped concurrency — and you inflate queue time and wall-clock, which multiplies across every engineer waiting. An organization’s engineer-hours dwarf its CI bill, so the true optimum is usually more parallelism, not less: add runners until wall-clock stops improving, because past that point you’re paying for machines that don’t shorten the wait, but up to that point every minute shaved is multiplied by the team.

Two levers make “more parallelism” cheap. First, spot / preemptible instances: CI is fungible and retry-safe — a preempted job just re-runs — so it’s the ideal workload to put on spot, cutting the runner bill 60-90%. Second, right-sizing: most jobs need a small runner; reserve the big machines for the few jobs that actually get faster on them. A README lint on a 64-core runner is pure waste.