The #1 Mistake in File Upload Design
44sExposes a common bottleneck that resonates with developers and sparks debate.
▶ Play ClipAI Summary
This video walks through the step-by-step design of a file upload service, starting from a simple architecture and progressively evolving it into a scalable, secure, and reliable solution. The presenter evaluates the initial design against functional requirements, identifies bottlenecks, and introduces components like object storage, message queues, and CDNs to address them.
Chapters
[00:02]
Initial Requirements
The service must support file upload/download, handle large files, ensure security, and allow async processing for background tasks.
[00:46]
Simple Architecture
A basic design includes clients, a file upload API, and a relational database storing both files and metadata. This is a primitive implementation.
[01:25]
Bottlenecks Identified
The simple design fails the 'large files' requirement because the API becomes a bottleneck and files are loaded into memory. Storing binary data in a database causes page size explosion.
[02:29]
Introducing Object Store
To solve file storage issues, an object store like S3 or Azure Blob Storage is added. This handles large files efficiently and supports multipart upload, pause/resume, deduplication, and retention control.
[02:56]
Pre-signed URLs
The API now provides a pre-signed URL endpoint, allowing clients to upload files directly to the object store, removing the API as a single point of failure and maintaining security through authentication.
[04:17]
Async Processing
Background tasks like virus scanning, thumbnail generation, OCR, and validation are handled via a message queue. The object store can either queue a message after upload or send a callback to the API.
[05:28]
Download Optimization
A CDN is introduced for public/static files, routing requests to the geographically closest edge. Private files are served via pre-signed download URLs from the object store.
[06:44]
Final Design Review
The final architecture meets all requirements: upload/download, metadata storage, large file support, security controls (public/private/shared), and async processing.
[07:11]
Self-hosting Options
Open-source object stores like RustFS, SeaweedFS, and MinIO are S3-compatible and can be self-hosted if cloud solutions aren't preferred.
The video demonstrates a practical, iterative approach to system design, showing how to evolve a simple solution into a robust architecture by leveraging specialized components like object stores, queues, and CDNs. It emphasizes evaluating trade-offs and meeting functional requirements at each stage.
Mentioned in this Video
Tutorial Checklist
5
04:17
Set up a message queue and background workers for async processing (e.g., virus scanning, thumbnails).
Study Flashcards (5)
What are the four functional requirements for the file upload service?
easy
Click to reveal answer
What are the four functional requirements for the file upload service?
easy
Click to reveal answer
Upload and download files, work with large files, secure the system, and allow async processing.
00:17
Why is storing files directly in a relational database problematic?
medium
Click to reveal answer
Why is storing files directly in a relational database problematic?
medium
Click to reveal answer
It causes page size explosion because large files span multiple database pages, leading to performance issues.
02:04
What is the purpose of a pre-signed URL?
medium
Click to reveal answer
What is the purpose of a pre-signed URL?
medium
Click to reveal answer
It provides a secure URL that allows clients to upload or download files directly to/from the object store, bypassing the API.
02:56
Name two open-source, S3-compatible object stores mentioned.
easy
Click to reveal answer
Name two open-source, S3-compatible object stores mentioned.
easy
Click to reveal answer
RustFS and SeaweedFS (also MinIO).
07:11
How does the system handle async processing?
hard
Click to reveal answer
How does the system handle async processing?
hard
Click to reveal answer
The object store enqueues a message after upload, or sends a callback to the API, which then sends a message to a queue consumed by background workers.
04:44
💡 Key Takeaways
🔧
Object Store as a Specialized Solution
Introducing a dedicated object store solves the database page size issue and adds features like multipart upload and deduplication.
02:29
⚖️
Pre-signed URLs for Security and Scalability
This pattern removes the API as a bottleneck while maintaining security through authentication.
02:56
🔧
Async Processing via Message Queues
Decoupling background tasks with a queue allows for scalable processing of virus scans, thumbnails, and OCR.
04:17
💡
CDN for Public Files
Using a CDN improves performance for static content by serving from geographically close locations.
05:28
📊
Self-hosting Options
Open-source alternatives like RustFS and SeaweedFS provide flexibility for teams avoiding cloud lock-in.
07:11Full Transcript
[00:02] to walk you through a system design discussion where we are going to build a file upload service step-by-step starting from a very simple scenario and then expanding it into a more reliable and scalable solution. So, let's start
[00:17] by listing out some requirements for our file upload service. These are going to be our functional requirements and we need to be able to upload and download files, obviously. We need to be able to work with large files. We need to make
[00:32] our system secure and we also have to allow for async processing of the files long with any background work that we have to perform behind the scenes. So, we may start simple where you have your clients and they have some files that
[00:46] let's add a component here. This is going to be our file upload API and this is going to be their entry point for their initial request. So, they're going to send the request to our API and we
[00:59] could have an endpoint like files and then {slash} upload, for example. We're also going to need some database and this is going to hold our files and metadata for our initial implementation. It doesn't really matter which specific
[01:11] some sort of relational database. So, our API needs to be able to store the could be our initial primitive implementation. Now, let's look at our functional requirements. So, we'll be able to upload and download files, store
[01:25] our metadata. Now, the ability to work with large files is somewhat questionable as all of the requests are flowing through our API. So, it's going to quickly become a bottleneck if many users are attempting to upload a file at
[01:39] file in memory before we can store it in our database. So, I'm going to leave a should probably fail this requirement. When it comes to security, we can still secure this just fine and then async processing is somewhat possible with
[01:52] still running on our single API instance, then we're going to quickly run into the same bottlenecks. Another problem that we're going to run into is with the fact that we are initially storing the files and the metadata in
[02:04] our database. So, most databases can store binary data just fine, but this is going to cause problems with page size explosion as we can assume these files can be large and most databases have an upper limit on how large a single page
[02:16] can be. So, you can imagine a single file spanning multiple database pages to the database and also querying from the database. So, it's something to keep in mind and this is where we could reach out to our more specialized solution for
[02:29] working with files. So, we can introduce another component into our system and we're going to call this the object store. So, these are specialized couple examples. So, let's say something like S3 or Azure blob storage and
[02:42] they're designed to work with files efficiently. So, with the object store slightly alter how clients are interacting with the file upload API. So, what's going to happen is instead of having an endpoint to directly upload
[02:56] introduce a different API endpoint. So, let's add it here and let's call it something like files pre-signed upload. So, the idea here is we get what's called a pre-signed URL and it's supported by most of these object stores
[03:10] and this gives you a secure URL that allows you to send a request directly to the object store. So, after getting the pre-signed URL, our client is then going to send another request which is going to upload the file directly to the
[03:24] big bottlenecks inside of our system. So, first of all, the file upload API is no longer a single point of failure as we can now upload and download files from the object store using pre-signed URLs, which means we still have security
[03:38] request to the upload API, which means they have to be authenticated in order to even get a pre-signed URL and then we can get the benefits of object stores where they can work with large files. They We have support for multipart
[03:51] upload, which means we can break up a large file into multiple chunks and then stream those to the object store. This also gives us the ability to pause and resume uploads. We also get support for deduplication and we also get support
[04:05] for controlling file retention. So, instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API. Another component we need to support is the ability to have async
[04:17] processing. So, let's say we need to do a couple of operations behind the scenes to add a couple of components here. So, let's say we need to perform virus scanning, then we could have things like thumbnail generation or more general
[04:31] preview generation. We could perform things like optical character recognition or something more general like validation. So, all of these can be especially when we need to support a large number of file uploads. So, how do
[04:44] these operations? Well, we need to introduce another component into our system. So, let's add some sort of pipeline and this is going to be our database. And then we have a couple of ways how we could integrate this. Either
[04:58] we could have our object storage and queue a message after upload and this our background services as simple message consumers or we could have our object storage send some sort of callback to our file upload API and then
[05:13] our file upload API would be responsible for sending a message to the queue on approach here is probably more scalable as already have support for this built popular messaging systems. So, let me actually move all of these components
[05:28] over here as I'm going to need the bottom part of the screen for one more thing we want to add and that is after mostly solving the file upload part, we also have to solve how to reliably and securely allow our users access to these
[05:40] option to download the files. So, for this we can introduce what's It's a content delivery network where our clients can send a request to download a file, and if it's not available, the content delivery network can reach out
[05:53] to our API to provide this file. And this is mostly good for public or static files. So, you can think of things like documentation that can be publicly accessible, and things that don't change that often. Of course, static assets for
[06:06] your website. And why a content delivery network is good is because it's usually your users' request to the geographically closest component. Now, what about requests for some files? Those we can route directly to our API.
[06:19] And to implement this securely, we're also going to utilize pre-signed URLs. Except, this is going to be some sort of download endpoint, for example. And this is going to follow a similar idea as uploading to the object store, and it's
[06:31] the power of our object store to get access to their files. So, this final design should now check most of our boxes. We can upload and download file, we can store the metadata either in the object store or inside of our custom
[06:44] after a callback, or we can consume the message from a queue. We can work with large files, as most object stores support this. We also have security control for each file. We can make them public, we can make them private, we can
[06:58] share them between users, and we've also got the ability to do async processing, from some background worker, and perform the work behind the scenes, and then notify our file upload service when the processing is done. And lastly, I just
[07:11] want to leave off with an idea. If you want to self-host an object store, there are open source options out there, like for example, RustFS, which is fully S3 compatible, and you could run it on your own system if for some reason you don't
[07:23] want to use a cloud solution like AWS S3 or Azure Blob Storage. And another option I ran into is SeaweedFS. There are options out there like MinIO, which RustFS or SeaweedFS. And I'm going to
[07:36] the description of this video. Let me know in in comments what you would change about this design and what other considerations you would have made that would like to see more system design discussions like this one, consider
[07:50] leaving a suggestion for what topic I should cover. If you enjoyed this video, gently tap the like button to let me know. Thanks a lot for watching and know. Thanks a lot for watching and until next time, stay awesome.
