Stop wasting server resources on file uploads
45sHighlights a common mistake and offers a simple, powerful fix that saves time and resources, appealing to developers and interview candidates.
▶ Play Clip"Delivers exactly what the title promises: a clear explanation of a common mistake and its solution, with no fluff."
This video explains a common mistake in system design interviews: uploading large files through an API server to S3, which wastes resources. It presents the solution of using pre-signed URLs to allow direct client-to-S3 uploads, improving efficiency and scalability.
Candidates often design a system where the client uploads a file to the API server, which then forwards it to S3. This results in paying for the transfer twice and tying up server resources.
A 2GB video upload holds a server thread hostage for the entire transfer. With 10,000 concurrent uploads, servers become overwhelmed with moving bytes instead of doing application work.
The client requests an upload slot from the API. The server generates a time-limited, pre-authorized URL pointing directly to S3 and returns it to the client.
The client uploads directly to S3, bypassing the server. Once S3 finishes, it can trigger a completion event to start downstream tasks like transcoding or metadata updates.
The API now handles only a few hundred bytes of JSON instead of gigabytes of binary data, which is the appropriate workload for an application server.
Using pre-signed URLs for direct client-to-S3 uploads is a scalable and efficient pattern that avoids wasting server resources, making it a key concept for system design interviews.
What is the common mistake in system design when uploading large files?
Sending the file to the API server and having it forward to S3, which wastes resources.
What problem occurs when a 2GB video upload is sent through the API server?
It holds a server thread hostage for the entire transfer.
00:13
What is a pre-signed URL?
A time-limited, pre-authorized URL pointing directly to S3, allowing direct client uploads.
00:25
How does the client upload a file using a pre-signed URL?
The client uploads directly to S3 using the pre-signed URL, bypassing the server.
00:37
What happens after S3 finishes the upload?
S3 fires a completion event to trigger downstream tasks like transcoding or metadata updates.
00:50
Double Payment for Uploads
Highlights a costly inefficiency that many candidates overlook.
Server Thread Hostage
Illustrates the severe performance impact of relaying large files.
00:13Pre-signed URL Solution
Introduces a key technique for scalable file uploads.
00:25Event-Driven Downstream Processing
Shows how to decouple upload from processing for better scalability.
00:50[00:00] The client uploads a video to your server. Your server then uploads it to S3. But be careful, you just paid twice for the exact same work. This is one of the most common mistakes I see in system design interviews. A candidate's building a system where they need to upload a big file,
[00:13] and their first instinct is to send that file to the API server and then have the server forward it to S3. On the surface, this works, but then it doesn't. A 2GB video upload holds a server thread hostage for the entire transfer.
[00:25] At 10,000 concurrent uploads, your servers aren't doing application work anymore. They're just moving bytes around and wasting memory. The fix is what's called a pre-signed URL. Your client just asks the API for an upload slot.
[00:37] Your server generates a time-limited, pre-authorized URL, pointing directly at S3 and hands it back. Now the client uploads straight to S3, and your server never touches that file. When S3 finishes, it can fire off a completion event,
[00:50] and then kick off all that downstream transcoding or updating a metadata or anything else you need to do. Your API went from handling gigabytes of binary data to handling just a few hundred bytes of JSON, which is the right job for an application server.
[01:03] Learn more to prepare for your interviews at hellointerview.com.
⚡ Saved you 0h 01m reading this? Transcribe any YouTube video for free — no signup needed.