Robust CLI scripts: exit codes, logging, retries, signals

Exit codes: the contract with the OS

Every automation system — cron, CI, Kubernetes, Makefiles — makes one assumption about your script: exit 0 means it worked, non-zero means it didn’t. When you break that contract, you break the whole alerting chain. A process reports success or failure with exactly one number: its exit code. 0 means success; anything non-zero means failure. That number is not decoration — it is the only thing cron, CI runners, && chains, systemd, and every orchestrator look at to decide whether the next step runs or an alert fires. backup.sh && upload.sh runs upload only if backup exits 0. CI marks a step red only on non-zero. So the exit code is the success signal, and getting it wrong is how a failed job reports green.

import sys, logging

log = logging.getLogger("export")

def main() -> int:
    try:
        data = fetch_export()      # may raise on a 500
        write_atomic("/data/export.json", data)
        return 0                   # success — and only here
    except Exception:
        log.exception("export failed")  # ERROR + traceback to stderr
        return 1                   # FAILURE is now visible to cron/CI

if __name__ == "__main__":
    sys.exit(main())              # the return value becomes the exit code

The deadly pattern is the inverse: catch the exception, print a message, and let main fall off the end. A function that falls off the end returns None; sys.exit(None) exits 0. You have just told the OS the job succeeded while it actually failed. Either let the exception propagate (Python exits non-zero with a traceback) or explicitly sys.exit(1). Reserve distinct codes (2, 3, …) only if a caller actually branches on which failure — otherwise 1 is plenty.

Logging, not print: the unattended-script rule

print writes to stdout with no level, no timestamp, and no routing control. In an interactive session that’s fine; in a cron job it’s invisible — there’s no terminal, and stdout is often captured as the job’s data or discarded. The logging module gives you what an unattended script needs: levels (DEBUG/INFO/WARNING/ERROR), timestamps, and a destination configured once. The two rules that matter: diagnostics go to stderr, real output goes to stdout (so a downstream | jq sees clean data, not your log lines), and the level is set from a flag or env var so you can dial up DEBUG without editing code.

import logging, os, sys

logging.basicConfig(
    level=os.environ.get("LOG_LEVEL", "INFO"),
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
    stream=sys.stderr,            # diagnostics to stderr; stdout stays clean for data
)
log = logging.getLogger("export")

log.info("starting export run")          # routine progress
log.error("upstream returned 503")       # a problem worth alerting on
log.exception("unexpected failure")      # ERROR + full traceback, inside an except block

log.exception is the one to reach for inside an except block — it logs at ERROR and attaches the traceback, which is exactly what you want shipped to your log aggregator. When you see a cron job with print statements for diagnostics, ask yourself: where does that output actually go, and who is reading it at 3am when the job fails? print-debugging a cron job produces nothing greppable; structured logs are searchable, shippable, and tie straight into observability.

Retries with backoff: surviving transient I/O

Network and API calls fail transiently: a timeout, a 503, a rate-limit. The robust response is not “give up” and not “retry forever” — it’s a bounded retry with exponential backoff and jitter. Wait ~1s, then ~2s, then ~4s, each with a random component so a fleet of scripts doesn’t retry in lockstep and stampede the recovering service, up to a cap of attempts. Two hard constraints: only retry idempotent or transient operations, and respect Retry-After when the server sends it.

import time, random, logging

log = logging.getLogger("export")

def with_backoff(op, *, attempts=5, base=1.0, cap=30.0):
    for i in range(attempts):
        try:
            return op()
        except TransientError as e:           # ONLY transient/retryable errors
            if i == attempts - 1:
                raise                          # budget exhausted -> let it fail
            delay = min(cap, base * 2 ** i) + random.uniform(0, 1)  # backoff + jitter
            log.warning("attempt %d failed (%s), retrying in %.1fs", i + 1, e, delay)
            time.sleep(delay)

The asymmetry is the whole point. A bounded backoff turns a transient 503 into a non-event — the third attempt succeeds and nobody notices. An unbounded or instant retry does the opposite: it hammers an already-struggling service, turning a blip into an outage. And retrying a non-idempotent POST (charge a card, send an email) can double-apply the side effect — so retry reads and idempotent writes, and make non-idempotent operations idempotent with a key before you retry them.

Signals and atomic writes: dying cleanly

An unattended process gets killed. The orchestrator sends SIGTERM to stop it; a human sends SIGINT with Ctrl-C. By default both unwind the stack — but if you’re halfway through writing a file, the default leaves a truncated file, a held lock, or a temp dir full of garbage. Two complementary defenses: handle the signal (or rely on try/finally + context managers) to flush and clean up, and never write the real file in place — write a temp file, then os.replace, which is atomic on a POSIX filesystem. A kill at any instant leaves either the old complete file or the new complete file, never a half-written one.

import os, signal, tempfile, json

def write_atomic(path: str, obj) -> None:
    d = os.path.dirname(path) or "."
    fd, tmp = tempfile.mkstemp(dir=d)          # same filesystem -> rename is atomic
    try:
        with os.fdopen(fd, "w") as f:
            json.dump(obj, f)
            f.flush()
            os.fsync(f.fileno())               # durable before the swap
        os.replace(tmp, path)                  # atomic: readers see old OR new, never partial
    except BaseException:
        os.unlink(tmp)                         # killed mid-write -> no orphan temp
        raise

def handle_term(signum, frame):
    raise SystemExit(143)                      # turn SIGTERM into a clean unwind (128+15)

signal.signal(signal.SIGTERM, handle_term)     # finally/with blocks now run on stop

Because os.replace is atomic, the kill window where a partial file could be observed simply doesn’t exist. Combine that with idempotency — a script will be re-run (a retry, a manual rerun, an overlapping cron tick), so check-before-act, write atomically, and take a lockfile to refuse a second concurrent run. An idempotent script you can safely run twice; a non-idempotent one corrupts state the second time.