S3 object storage: classes, lifecycle, presigned URLs, durability

Buckets, keys, and the flat keyspace

S3 is an object store, not a filesystem. An object is bytes plus metadata plus a key, and it lives in a bucket — a globally-named, region-scoped container. There are no real directories. The key logs/2026/06/04/app.log looks like a path, but it is a single flat string; the slashes are just characters. The console renders a folder tree as a convenience by splitting on /, but under the hood there is one giant lexicographic keyspace and nothing else. This matters because almost everything that scales in S3 scales by prefix — the leading portion of the key — not by folder, because folders don’t exist.

That flat model also shapes performance. S3 gives you at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix, and there is no cap on the number of prefixes. So request throughput is something you design by spreading keys across prefixes, not a fixed quota you hit. A bucket that funnels every write through one hot prefix (say, keys that all start with the same date) will throttle with 503 SlowDown long before the bucket itself is “full” — because a bucket has no size limit, only per-prefix request limits.

# A key is one flat string; the "folders" are just slashes in it.
aws s3api put-object \
  --bucket app-prod-events \
  --key "logs/2026/06/04/ingest-7f3a.json.gz" \
  --body ingest-7f3a.json.gz \
  --storage-class STANDARD

# List by prefix — this is what "ls a folder" actually does under the hood.
aws s3api list-objects-v2 \
  --bucket app-prod-events \
  --prefix "logs/2026/06/04/" \
  --query "Contents[].{Key:Key,Class:StorageClass,Bytes:Size}"

Since December 2020, S3 also gives strong read-after-write consistency for free, on every request, in every region. After a successful PUT — whether it’s a brand-new object, an overwrite, or a delete — any later GET, HEAD, or LIST immediately sees the latest state. The old eventual-consistency caveats (“you might read a stale or missing object right after writing”) are gone. You no longer need read-your-write workarounds; if a read looks stale today, the bug is in your code or caching layer, not in S3.

Storage classes: the retrieval-versus-min-duration tradeoff

When you store data in S3, the default class is never neutral — it is a billing decision you make once and then pay for silently every month. Before you reach for the first class on the list, ask yourself: how often will this data actually be read, and how fast does it need to come back? The answer drives every tradeoff below.

Every object carries a storage class, and that single attribute is the main cost lever. The classes trade three things against each other: per-GB storage price, retrieval latency, and a minimum storage duration you pay for even if you delete early. Numbers below are from the AWS docs; dollar figures are illustrative and region-dependent — always price against the S3 pricing page.

S3 Standard is the default: stored across ≥3 Availability Zones, millisecond access, no minimum duration, no retrieval fee. It is the right home for hot data and the wrong home for anything cold — you’re paying premium storage for bytes nobody reads.

Standard-IA (Infrequent Access) keeps the same ≥3-AZ resilience and millisecond access but cuts the storage price, in exchange for a per-GB retrieval fee and a 30-day minimum duration. One Zone-IA is cheaper still because it stores in only one AZ — same durability design, but if that AZ is physically lost, the data is gone. Intelligent-Tiering sidesteps the guessing: for a small per-object monitoring fee it moves each object between access tiers automatically as access patterns shift, with no retrieval fee and no minimum duration — the safe default when you genuinely don’t know the pattern.

The Glacier family is for archives. Glacier Instant Retrieval keeps millisecond access (90-day min duration) for rarely-read data you still need fast. Glacier Flexible Retrieval (90-day min) and Glacier Deep Archive (180-day min) are true cold storage: objects are archived and not available for real-time access — you must issue a RestoreObject and wait minutes to hours (Flexible) or hours (Deep Archive) before you can read them, and you pay a per-GB retrieval fee on top. That retrieval fee and latency are exactly what the Hook’s team walked into.

Lifecycle policies and presigned URLs

You rarely move objects between classes by hand. A lifecycle configuration on the bucket does it on a schedule: transition objects to a colder class after N days, and expire (delete) them after M days. This is how you encode “hot for a month, infrequent for a quarter, archived for a year, gone after seven years” as one rule instead of a cron job.

{
  "Rules": [
    {
      "ID": "events-cooling",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 90,  "StorageClass": "GLACIER" },
        { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
      ],
      "Expiration": { "Days": 2555 }
    }
  ]
}

The second workhorse is the presigned URL. By default every object is private — only the owner can read it. A presigned URL embeds a time-limited, cryptographic signature (SigV4) so anyone holding the URL can perform one specific operation — a GET to download or a PUT to upload — until it expires. The signature carries the permissions of whoever generated it, so it can never grant more access than that signer already has. The standard pattern: your server signs a PUT URL and hands it to the browser, which uploads the file straight to S3 — your application servers never touch the bytes, so a 2 GB upload doesn’t tie up an app process or count against your egress. The expiry is the whole security model: an SDK/CLI presigned URL can live up to 7 days (604,800 seconds); the console caps at 12 hours. Sign short.

import boto3
s3 = boto3.client("s3")

# A time-limited PUT URL: hand this to the browser so it uploads
# directly to S3. Expires in 15 minutes; carries the signer's perms.
upload_url = s3.generate_presigned_url(
    ClientMethod="put_object",
    Params={"Bucket": "app-prod-uploads", "Key": "user/42/avatar.png"},
    ExpiresIn=900,  # seconds; max 604800 (7 days) for SigV4
)