Media & storage: build a chunked, deduplicated upload-and-sync service

Build a small file upload-and-sync service that stores files as content-addressed chunks in an object store (real S3/MinIO or a local blob directory), keeps metadata in a separate database, syncs edits by transferring only changed chunks, deduplicates identical chunks, and resolves conflicting edits via a version chain — then measure the bandwidth and storage wins your design produces.

• Stand up an object store (MinIO, S3, or a local directory acting as a blob store) and a separate metadata database (SQLite/Postgres). Define the split clearly: chunks (immutable bytes, keyed by content hash) live in the object store; file metadata (name, version, ordered chunk-hash list) lives in the DB.
• Implement upload so bytes go straight to the blob store, not through an in-memory buffer or a DB column: split the file into fixed-size chunks (e.g. 1–4 MB), hash each (SHA-256), upload only chunks whose hash isn't already stored, then commit a new file version listing the chunk hashes — chunks first, metadata second.
• Implement download/read by looking up the version's chunk list in metadata and streaming the chunks from the blob store in order; put a small cache (in-memory or Redis) in front of the metadata lookups (not the chunk bytes).
• Implement delta sync: given an edited file, re-chunk it, diff its chunk hashes against the previous version, upload only the new-hash chunks, and commit a new version — log how many bytes you uploaded vs the file size.
• Implement deduplication and prove it: upload the same file (or a shared chunk) under two different names/users and show only one physical copy is stored; keep a reference count so a chunk is deleted only when its last referencing version is gone.
• Implement conflict resolution: have each version record its parent. When a sync arrives whose parent is no longer the current head (simulate two clients editing from the same base), detect the divergence and keep BOTH as conflicted copies instead of overwriting — never lose an edit.
• Add a tiny multipart-style upload for one large file: split into parts, upload parts independently with one part deliberately failing and retried alone, then 'complete' to assemble — show the object becomes visible only after completion.
• Measure and report: for a representative edit (e.g. change one chunk of a 100 MB file), report bytes moved by delta sync vs a naive full re-upload, and the storage saved by dedup across duplicate uploads — to one significant figure.

• A working service where an upload stores chunks in the object store and a metadata row in the DB, with NO file bytes ever held in a DB column or buffered whole in app memory — demonstrate this with a multi-hundred-MB file.
• A delta-sync demonstration: editing one chunk of a large file uploads only ~one chunk's worth of bytes, with a logged before/after byte count showing the order-of-magnitude saving vs full re-upload.
• A dedup demonstration: the same content stored twice occupies one physical chunk (shown via the blob store's object count and the reference count), and deleting one referencing version does not delete the still-referenced chunk.
• A conflict demonstration: two edits from the same parent version both survive as conflicted copies, with the version chain (parent pointers) shown — a last-writer-wins overwrite must NOT occur.
• A short write-up: the metadata-vs-blob split you chose and why, the measured delta-sync and dedup wins to one significant figure, and one paragraph on what would change to reach S3-like durability (erasure coding across failure domains) at real scale.