backend · intermediate · 4d
Presigned upload flow
Direct-to-storage uploads via presigned URLs with size/content-type limits and a completion webhook that verifies the object actually arrived — so your API server never touches file bytes.
Deliverable
An upload flow where the client gets a presigned PUT URL, uploads directly to S3-compatible storage, and the server confirms receipt by fetching object metadata — never proxying file bytes.
Milestones
0/2 · 0%- 01Issue a constrained presigned PUT
Issue a presigned PUT URL from the API with an expiry, allowed content-type header, and a max Content-Length constraint.
Definition of done- The API returns a presigned PUT URL with a short expiry, a fixed allowed content-type, and a max Content-Length.
- An expired or tampered URL (wrong content-type / oversize) is rejected by storage, not by the API.
- 02Direct upload + verified receipt
Configure CORS on the bucket so the browser can upload directly without a proxy, then verify the object arrived by checking its ETag and size in the webhook.
Definition of done- The browser uploads directly to the bucket (CORS preflight passes) with no proxy through the API.
- The completion webhook confirms the object arrived by fetching its size/ETag, and is idempotent under duplicate deliveries.
Rubric
| Junior | Mid | Senior | |
|---|---|---|---|
| Presigned URL constraints | The API issues a presigned PUT with no constraints; a client can upload a 5 GB video where a 2 MB image was expected, or substitute an arbitrary content-type. | The URL is signed with an expiry, a fixed allowed Content-Type, and a max Content-Length; a tampered or oversize PUT is rejected by storage, not by the API, so enforcement is in the signature rather than in per-request application logic. | You address key-clobber abuse: a presigned PUT to a predictable key allows any authenticated user to overwrite another user's file. You fix this with server-assigned UUID keys (never caller-chosen) and reason about the race window between upload and completion — a client can upload a valid file, then replace it before the webhook fires by re-using the URL within its expiry. |
| CORS & direct-upload flow | The browser upload is proxied through the API server; every upload passes through application memory and blocks a request thread for the duration. | The S3 bucket CORS policy allows the browser origin to PUT directly; a preflight OPTIONS request passes cleanly, and no file bytes touch the API server. | You lock the CORS policy to the minimum necessary: the exact allowed origin(s), the specific headers needed (Content-Type, x-amz-*), and a short max-age for the preflight cache. You explain why a wildcard CORS policy on a public bucket is not a vulnerability (no cookies, no ambient authority) but why it is still wrong on a private bucket where presigned URLs carry the authorization. |
| Completion webhook & receipt verification | The client self-reports completion; the server trusts the report and marks the upload as received without verifying the object exists or matches what was requested. | The completion webhook fetches the object's ETag and size from storage and confirms they match the expected values; the webhook is idempotent so a duplicate delivery from a retry does not double-process. | You reason about the substitution window: between the presigned PUT expiry and the webhook firing, a race exists where a different file can be uploaded to the same key. You prevent this by storing the expected ETag (derived from the pre-upload contract) and refusing to mark a receipt as valid if the stored ETag differs — and you document the tradeoff: ETag verification catches substitution but breaks multipart uploads where the ETag is assembled from part hashes. |
Reference walkthrough (spoiler)
Why presigned PUT instead of proxying: proxying every upload through the API server ties one request-handler thread per upload for the full transfer duration. At 10 MB/s and a 100 MB file, that's 10 seconds of thread time per upload — a single server handling 100 concurrent uploads spends 1000 thread-seconds in I/O. Presigned URLs shift that I/O directly to object storage, which scales horizontally at negligible cost.
Content-type and size enforcement must be in the signature, not in the webhook: a webhook fires after the bytes are already in storage. If the constraints are checked only there, a malicious client can store a script disguised as an image and the application has already accepted it into its bucket. Encoding content-type and max-size in the presigned URL signature means storage itself enforces the policy before a single byte is written.
Server-assigned keys prevent clobber attacks: if the caller chooses the object key (e.g. their own username), they can overwrite any file they've previously uploaded or guess another user's key. A UUID generated by the server at presign time, stored in the pending-upload record, and verified in the webhook makes the key unguessable and the upload non-replayable to a different slot.
The completion race and ETag verification: the presigned URL is valid for its full TTL after it is issued. A client can upload a legitimate file, receive a passing ETag from the webhook, and then upload a different file to the same key within the URL's remaining TTL. Storing the expected ETag at presign time and refusing to accept a receipt that does not match it closes this window — at the cost of incompatibility with multipart uploads, which produce a composite ETag.
Make it senior
- Replace single-part PUT with S3 multipart upload for files over 5 MB: create, upload parts in parallel, and complete — with resumable state stored server-side.
- Add server-side virus scanning on the completion webhook using a Lambda/Worker trigger before marking the upload as safe.