[00:02] to walk you through a system design discussion where we are going to build a file upload service step-by-step starting from a very simple scenario and then expanding it into a more reliable and scalable solution. So, let's start
[00:17] by listing out some requirements for our file upload service. These are going to be our functional requirements and we need to be able to upload and download files, obviously. We need to be able to work with large files. We need to make
[00:32] our system secure and we also have to allow for async processing of the files long with any background work that we have to perform behind the scenes. So, we may start simple where you have your clients and they have some files that
[00:46] let's add a component here. This is going to be our file upload API and this is going to be their entry point for their initial request. So, they're going to send the request to our API and we
[00:59] could have an endpoint like files and then {slash} upload, for example. We're also going to need some database and this is going to hold our files and metadata for our initial implementation. It doesn't really matter which specific
[01:11] some sort of relational database. So, our API needs to be able to store the could be our initial primitive implementation. Now, let's look at our functional requirements. So, we'll be able to upload and download files, store
[01:25] our metadata. Now, the ability to work with large files is somewhat questionable as all of the requests are flowing through our API. So, it's going to quickly become a bottleneck if many users are attempting to upload a file at
[01:39] file in memory before we can store it in our database. So, I'm going to leave a should probably fail this requirement. When it comes to security, we can still secure this just fine and then async processing is somewhat possible with
[01:52] still running on our single API instance, then we're going to quickly run into the same bottlenecks. Another problem that we're going to run into is with the fact that we are initially storing the files and the metadata in
[02:04] our database. So, most databases can store binary data just fine, but this is going to cause problems with page size explosion as we can assume these files can be large and most databases have an upper limit on how large a single page
[02:16] can be. So, you can imagine a single file spanning multiple database pages to the database and also querying from the database. So, it's something to keep in mind and this is where we could reach out to our more specialized solution for
[02:29] working with files. So, we can introduce another component into our system and we're going to call this the object store. So, these are specialized couple examples. So, let's say something like S3 or Azure blob storage and
[02:42] they're designed to work with files efficiently. So, with the object store slightly alter how clients are interacting with the file upload API. So, what's going to happen is instead of having an endpoint to directly upload
[02:56] introduce a different API endpoint. So, let's add it here and let's call it something like files pre-signed upload. So, the idea here is we get what's called a pre-signed URL and it's supported by most of these object stores
[03:10] and this gives you a secure URL that allows you to send a request directly to the object store. So, after getting the pre-signed URL, our client is then going to send another request which is going to upload the file directly to the
[03:24] big bottlenecks inside of our system. So, first of all, the file upload API is no longer a single point of failure as we can now upload and download files from the object store using pre-signed URLs, which means we still have security
[03:38] request to the upload API, which means they have to be authenticated in order to even get a pre-signed URL and then we can get the benefits of object stores where they can work with large files. They We have support for multipart
[03:51] upload, which means we can break up a large file into multiple chunks and then stream those to the object store. This also gives us the ability to pause and resume uploads. We also get support for deduplication and we also get support
[04:05] for controlling file retention. So, instead of redesigning something like S3 from scratch, we can simply leverage it as another component inside of our file upload API. Another component we need to support is the ability to have async
[04:17] processing. So, let's say we need to do a couple of operations behind the scenes to add a couple of components here. So, let's say we need to perform virus scanning, then we could have things like thumbnail generation or more general
[04:31] preview generation. We could perform things like optical character recognition or something more general like validation. So, all of these can be especially when we need to support a large number of file uploads. So, how do
[04:44] these operations? Well, we need to introduce another component into our system. So, let's add some sort of pipeline and this is going to be our database. And then we have a couple of ways how we could integrate this. Either
[04:58] we could have our object storage and queue a message after upload and this our background services as simple message consumers or we could have our object storage send some sort of callback to our file upload API and then
[05:13] our file upload API would be responsible for sending a message to the queue on approach here is probably more scalable as already have support for this built popular messaging systems. So, let me actually move all of these components
[05:28] over here as I'm going to need the bottom part of the screen for one more thing we want to add and that is after mostly solving the file upload part, we also have to solve how to reliably and securely allow our users access to these
[05:40] option to download the files. So, for this we can introduce what's It's a content delivery network where our clients can send a request to download a file, and if it's not available, the content delivery network can reach out
[05:53] to our API to provide this file. And this is mostly good for public or static files. So, you can think of things like documentation that can be publicly accessible, and things that don't change that often. Of course, static assets for
[06:06] your website. And why a content delivery network is good is because it's usually your users' request to the geographically closest component. Now, what about requests for some files? Those we can route directly to our API.
[06:19] And to implement this securely, we're also going to utilize pre-signed URLs. Except, this is going to be some sort of download endpoint, for example. And this is going to follow a similar idea as uploading to the object store, and it's
[06:31] the power of our object store to get access to their files. So, this final design should now check most of our boxes. We can upload and download file, we can store the metadata either in the object store or inside of our custom
[06:44] after a callback, or we can consume the message from a queue. We can work with large files, as most object stores support this. We also have security control for each file. We can make them public, we can make them private, we can
[06:58] share them between users, and we've also got the ability to do async processing, from some background worker, and perform the work behind the scenes, and then notify our file upload service when the processing is done. And lastly, I just
[07:11] want to leave off with an idea. If you want to self-host an object store, there are open source options out there, like for example, RustFS, which is fully S3 compatible, and you could run it on your own system if for some reason you don't
[07:23] want to use a cloud solution like AWS S3 or Azure Blob Storage. And another option I ran into is SeaweedFS. There are options out there like MinIO, which RustFS or SeaweedFS. And I'm going to
[07:36] the description of this video. Let me know in in comments what you would change about this design and what other considerations you would have made that would like to see more system design discussions like this one, consider
[07:50] leaving a suggestion for what topic I should cover. If you enjoyed this video, gently tap the like button to let me know. Thanks a lot for watching and know. Thanks a lot for watching and until next time, stay awesome.