Object Storage in System Design Interviews w/ Ex-Meta Staff Engineer

[00:02] definitely think of it as one. Think of it as a database that's designed specifically for large files. And these large files are commonly referred to as binary large objects or blobs. But in reality, all that they are are things

[00:18] like videos, photos, uh music files, JSON files, large text files. And each of these, if you were to actually open your terminal and look at, for example, a photo, what you would find is that they're just large collections of bytes,

[00:32] there's some oftent times in the order of megabytes amount of bytes that need to be stored somewhere. And that's where object storage comes into play. Now, the somebody's mind when they're learning about object storage for the first time

[00:46] is why don't we just store this in a quote unquote normal database? Well, the answer is that normal databases, where normal here means a relational or an OOLTP database, they're built for these small frequently changing records that

[00:59] need rich queries and things like joins. And so, they're not made to handle large, mostly static files like the ones that we looked at above. So, let me show you with this example cuz I think that this will be pretty illustrative.

[01:13] photos right alongside all of the other user information that's in your user to you. You have users, you're storing their roles, their names, all of these different things, and then additionally their profile image. But Postgress, we

[01:28] can use as our example of a relational database here, packs all of these rows internally into 8 kilobyte pages. And so image like we have here, that's going to span 500 whole pages. And now you might

[01:43] why is that so bad? Why is having 500 pages for an image bad? Well, I want you to imagine a simple query like this where you just want to pull in the top 50 users. Well, even though you only want a small piece of the data, having

[01:57] these large images stored alongside all of the user data is going to create a ton of extra overhead. The database has to manage and potentially access these huge files, these many megabytes of pages that we have now. And this is

[02:09] going to impact things like performance, increase memory pressure, slow down what increase memory pressure, slow down what should be uh fast and simple queries. need to consider replication as another issue. When you replicate data across

[02:23] multiple database servers for either backups or scaling, that four megabyte blob has to be copied to every single replica with each write, which is going to create a ton of lag and consume massive bandwidth. Then there's the

[02:36] notion of backups more specifically where these 4 megabyte images are going to be included in those maybe nightly backups that you run. And so you're going to turn then in a restoration process which should only take maybe a

[02:50] so you can picture yourself, you're working, you're up late, your database has gone down, gone down, your users are frustrated, and you're trying to restore from a backup. But your backup is huge because it's bloated with all of these

[03:02] large static files. And so the restoration of your database is going to take many hours. So the bottom line here is pretty straightforward. traditional databases, these relational or OOLTP databases, they choke on blobs both in

[03:16] cost. And so hopefully this has made it these directly in your traditional database, this is where object storage comes into play. You should be storing them strictly speaking in your object or

[03:28] ahead and take a quick look at what's happening under the hood of this diagram is a obviously a severe accurate and incredibly useful I think for for considering how this works but

[03:42] it is of course simple. The key is that all of your files are stored just on these cheap storage nodes. This is meaning that they are stored on disk in servers in some racks in some data warehouse somewhere. Now the question

[03:56] becomes how do we know where our file is? And so when a client wants to get or storage the first thing that they're going to do is that they're going to make a request to a metadata service. And this metadata service which is part

[04:09] of blob storage is part of object storage is responsible for using an index to look up where that file is located. And so in our case it's located one is. The metadata service will say oh server A holds file one. Now in reality

[04:23] servers. This is important for redundancy which we'll talk about in a second. But at least one copy of file one exists in server A. And so that's what we're going to do here. Now, server A's responsibility then is to stream

[04:37] this file, those bytes of this file back to the client. And this might be via a pre-signed URL or via the gateway. But in either case, we're going to stream directly from the server right back uh to the user. It's really that

[04:50] straightforward. It's a simple lookup, right? A connection and then a stream of the file back to the user. But the important bit is what makes this so And there's three things that I want to call out. The first is these flat name

[05:05] spaces. And so unlike your your local machines where you have to navigate through these these folder trees to find your files, object storage uses just a thinking to yourself, wait a minute, I've used S3 and there's buckets and

[05:19] there's kind of these folder slash patterns. That's all just UI sugar in order to make it easier for us as humans to use. In reality, under the hood, it's just storing that single string, which allows us to directly locate the file by

[05:33] name, uh, making it incredibly fast to look up. We don't have to navigate a tree. The second thing is is these immutable rights. And so, unlike databases where you're constantly updating records, uh, in this case, you

[05:46] can't modify files in object storage. You can't change bytes in the middle of the file. You can only create new versions, which is really common, or overwrite the file entirely. And so this is important because it eliminates the

[05:58] needs within the the database engine for locks and race conditions which is going to make things a lot simpler, faster, and ultimately cheaper. And then the last one to call out here is redundancy. And so I talked about how file one in

[06:11] our case doesn't only exist on server A, right? It might also exist on server B or server C. And the reality is that every single object, every single file is either fully replicated or something called eraser coded across multiple

[06:24] different servers, multiple different racks, oftentimes even across multiple different data centers in cases. And we this is in order to achieve what we call this is in order to achieve what we call in the industry as 11 9 of durability.

[06:37] in the industry as 11 9 of durability. And so it means that this is durable 99.999 whatever percent durability. If you lose one node, no big deal. The data

[06:51] is still there. We can go fetch it from another node automatically and heal up in the background. That ultimately is the short and sweet of the high level here. So, let's keep moving and let's zone in on three things in particular

[07:03] know in the context of your system design interview. And so, the first one might seem really obvious to you at this point. It's that large files should be stored in object storage, but your metadata should be stored in your

[07:16] might seem obvious, but you'd be surprised. I've had quite a few junior candidates or me mid-level candidates in interviews keeping their metadata also in object storage. Um, strictly speaking, this isn't um, you know,

[07:30] impossible, but it's against the industry standard for for a number of here. You want the ability to quickly query for your metadata and then pull in those larger files just from object storage. So a really simple example here

[07:43] site. You have posts on that social media site. A user creates a post and text. Think of it like Twitter. And so we're going to store all of that in our database along with the link, the URL um

[07:57] to our file, the photo that's on this post, which is stored in object storage. return all this metadata and then they Talk more about that in a second. And so some examples here, photos, videos like

[08:11] I just mentioned for a messaging or a social app. Um user uploading files for Dropbox. You're going to store those files in object storage where the files metadata will be in the database. We also store static assets like um CSS and

[08:25] JavaScript that are needed in order to render our website. This gets stored in object storage. it ends up being fronted by a CDN as well, but don't worry too much about that if that's confusing. Uh, we store logs, log files, output log

[08:38] files from our application in S3 machine learning training data. This is a that you're going to use to train an LLM. Perfect thing to store in object Now, the second one here, and I kind of

[08:52] little bit, is that you can actually upload and download directly from S3 via what are called pre-signed URLs. And so, at least in the upload case, if you want to upload a file, you might think, I'm going to upload that file to my server,

[09:07] correct permissions, is going to upload directly to object storage. But the extra bandwidth. You have to go through your server first and then go to object store. And if your server um you know isn't particularly well scaled, it might

[09:20] not be able to handle incredibly large files, especially if there's an influx of them. It's wasteful. And so what typically happens in the industry is request from your object store what's called a pre-signed URL. And this is

[09:33] just going to be a URL that your client can use to put HTTP put basically to upload that file directly to the object store. So we're going to ask permission. We're going to say, I want to upload a file of this size, of this type. Can you

[09:47] give me a URL that's available for the next 30 minutes or an hour or so that I permission to upload directly to you. This way, the client gets a direct line waste the bandwidth going through here. The same story on downloads. So on

[10:01] downloads, we can get the S3 URL. Whether it's signed or not depends on your permissions, but in any case, you can just download or stream that file having to go through your server. This comes up a lot in system design

[10:14] interviews. And then the third one here is that large files are going to be uploaded in chunks and then we can stitch them back together. And so this is something that's called multi-art upload. Traditionally, at least S3 calls

[10:28] it this. But the important thing to note here is that there's limitations on the here is that there's limitations on the size of a file that can be posted on the internet, right? An HTTP post or put. And this is limitations sometimes in the

[10:41] browser, in gateways, in the servers themselves, they exist all over. Um, in S3's case, the most popular object storage, which we'll talk about here in a second, their limit is 5 megabytes, I believe. Don't quote me on that. I'm

[10:54] sure folks in the comments will tell me if I made a mistake there, but about 5 megabytes. And so what that means is that if we wanted to upload a, you know, 10 gigabyte file, even a 1 gigabyte file, we're going to have to chunk it

[11:06] into these 5 megabyte chunks and then upload each of those chunks. It could be in parallel, they could be sequentially, but the client's going to take the file, chunks, upload each of those chunks to the object store, and then our object

[11:20] storage is going to stitch those chunks together back into a full file once it has everything. And this is called multi-art upload again. Now, maybe the last thing to show here, what are some examples of popular object

[11:33] or blob stores that exist in the industry? By far, these are the three main ones. Amazon S3, this is the most popular in your interview. If you've S3. If you want to learn a little bit about one of them, S3 is the most

[11:46] into. But, of course, Google Cloud has their own. Google Cloud Storage, uh, Microsoft Azure has Azure Blob Storage. They all work very similarly. They all have these notions of pre-signed URLs, signed URLs for downloading. They all

[12:00] to know. Awesome. Okay. Um, tried to make that quick. It's a highle overview. This is the quick nitty-gritty of what you interviews. Hopefully, this was useful. If you enjoyed this, we have a lot more

[12:16] content like this that is high level, as well as tons of content that goes really deep on specific technologies and specific system designs over at hellin.com. If you haven't been over there, check it out. The overwhelming

[12:28] majority of the content is free and I think you'll really enjoy it. Thanks for watching and best of luck with your upcoming interviews. Till next time.