DynamoDB Deep Dive with an Ex-Meta Staff Engineer

[00:01] we're going to go into a deep dive on Amazon's Dynamo DB specifically through the lens of how and when to use it in a system design interview so we're going to go over the data model we're going to go over indexing talk about how to

[00:13] architecture as it pertains to scalability and availability uh we're features like dynamodb accelerator as well as Dynam DB streams and then we'll wrap up with most importantly when should you use it in an interview and

[00:27] in it's going to be a lot fun let's get after head around the the data model for Dynamo DB here there's three key

[00:39] know as it pertains to the Dynamo DB data model uh the first is a table so all data is stored in collections called tables this is exactly the same as with SQL these are just collections of related data but then within each table

[00:54] just individual records each with a unique primary key and these are like rows in a SQL table or rows in a spreadsheet and then within each item and those attributes are like columns in a spreadsheet or column in a in a SQL

[01:10] table but they're the actual data fields within each item like the name or the age and so we can take a look at this example that we have here this is a basic user table that you might have for any standard application and the table

[01:23] has a collection of users each particular item or row here is a given user and then the attributes are these columns the information about any given user and so one thing that's important to note here is that not all users have

[01:38] to note here is that not all users have all attributes and this is okay so Dynamo DB is schemes uh this means that items in the same table can have different attributes so we could even have a a new attribute here call it I

[01:51] don't know employer and only give that to Sarah and Jimmy but not to Stefan and Evan here and that would be okay um this is very different like I said than SQL where you have to conform to a table's rigid schemas with predefined columns

[02:07] and so there's pros and cons to this right on one hand it's fast and it's have to change our data structures there's no migrations um plus we only empty columns that take up space like we would in SQL which would just have

[02:21] undefined values but there's the trade-off our application now needs to handle missing attributes uh and we need to analyze data across items it becomes trickier we can't rely on the database to enforce the data consistency like we

[02:34] can in SQL and so like anything else there's pros and cons and potentially to weigh in your interview when you're making your

[02:46] in SQL databases we have a primary key that uniquely identifies each rows and helps us with those fast lookups and so that's what the user ID typically maybe would be here and why it's colored green now in Dynamo DB it's simple but instead

[02:59] we have two two things we have a partition key and a sort key and these lookups and data organization and so let's talk about about each of these the partition key is unique identifier that determines the item's physical location

[03:14] partition key we hash it using a consistent hashing function and then this determines which physical node which physical piece of Hardware this data lies on and then for sort Keys these are optional you don't need a sort

[03:28] key but it's a secondary attribute and it enables ordering and range queries within a partition and so for each sort key we end up having a b tree index you just think of a tree that's kept in memory uh and we use this for these

[03:40] all items between a start and an end time or when we want to sort for example like get all posts in a descending order uh based on the creation time and then the primary key is just a combination of these two things and so it's a

[03:55] combination of the partition key and optionally the sort key the reason that this matters is that in your system design interview if you're using Dynamo DB you're typically going to introduce your data model and explain

[04:08] to your interviewer what is your partition key and what is optionally your sort key and so if we're thinking about a chat app well we want our primary key oh excuse me this should say partition key we want our partition key

[04:22] to be the chat ID this means that all messages for a particular chat group chat are going to be collocated on the same node on the same physical Hardware bunch of different nodes and Scatter Gather in order to get all of the chats

[04:37] chat ID so that makes sense and then the typical operation is that we want to sort these by time now what we can do then is have the sort key as a message thinking you may be thinking why is it a message ID and not a timestamp well the

[04:54] you had two messages that were created at the same time and their primary key is a concatenation of the chat ID and the time they were created well these two things wouldn't be unique it would be a failed right and so instead we can

[05:08] have the same properties of having the sort key basically be Time by having the sort key basically be Time by having the message ID be monotonically increasing this is just a fancy way of saying that numbers that always go up and so each

[05:22] the last one and what this means is that when you sort by these IDs alphabetically you're automatically sorting by them uh by the time that they were created right and so for a chat application our partition key is a chat

[05:35] ID our sort key thus is the message ID for e-commerce maybe our most common query pattern is that we want to get all orders for a particular user and so we'd make our partition key user ID again to collocate all of the orders for a

[05:48] particular user on the same Shard or the same partition and then maybe we want to our Sor key as the order ID again a monotonically increasing ID and then for social media it's the really similar concept maybe we our most common query

[06:02] user and we want to be able to sort those by time so we have the exact same setup now these partition keys and sort keys are really effective in ensuring sort data but what if you needed to efficiently query by something other

[06:17] case maybe instead of efficiently also in a different part of my application need to efficiently query application need to efficiently query for all messages sent by a given user

[06:30] by something other than your sort key so instead of monotonically increasing uh message ID so sorting by time basically what if I want to sort by the number of attachments in messages well this is where secondary indexes come into play

[06:45] and so we have two types of secondary indexes we have one a global secondary index a GSI and two a local secondary index and so let's look at each of these a global secondary index is for that first use case where you want to be able

[06:58] to efficiently quer by something other than your existing partition key and so here's our main table we determined that our partition key was chat ID because the most common query pattern for us is to get all messages for a given chat um

[07:13] but we now introduced a new page on our website where we show users all of the chats that they've sent across chat IDs and it was super inefficient because we different partitions a bunch of different physical notes and so to solve

[07:28] this we create a global mobal secondary index on a different partition key in this case on the user ID and so now both of these things exist it's almost like a replica of the table and for our Global secondary index we can optionally

[07:43] project any of the attributes that we need here and so only the ones that we need in order to be be be U conscientious of our storage um would secondary index but now regardless of whether users are querying to see all

[07:57] messages of a chat or all messages sent by given user they can do so efficiently because we have this Global secondary index and then local secondary indexes this is for the case where we want to sort by something other than our

[08:11] existing sort key and so we have the same partition key in this case back to original partition key and the monotonically increasing message ID so we can sort by time uh as our sort ke key but now we want to also be able to

[08:26] sort by the number of attachments we have a a portion of our chat application lot of attachments being sent and we want to be able to see the messages that have the most attachments so that we can find attachments I don't know a bit

[08:39] contrived but you get the idea and so we can create a local secondary index where the sort key for that local secondary index is the number of attachments and so basically now we have a a b tree for message ID so that we can efficiently do

[08:52] range queries and sorts based on the message ID or the time and we also have that we can do the same relevant because if you find use cases where you have a uh two different

[09:05] functional requirements maybe that require different query patterns then that you're going to create a global secondary index or a local secondary index respectively and you can mention on what new Partition key or sort key in

[09:18] on what new Partition key or sort key in order to accomplish your a moment well you don't directly need to know this uh for an interview I always practically applied so how do you actually create a table in query with

[09:34] Dynamo DB well first off it's dead simple you go to the AWS console you table and it takes you here you specify your table name what your partition key is and your sort key um down here there's an additional button it's

[09:47] defaulted to leave the custom settings alternative or or the default settings settings and then specify some Global secondary indexes local secondary indexes you can also always build those after the fact so it's really that

[10:00] simple it's it's a couple button clicks what about with querying so if we come down here you can see what this looks like in SQL and you're probably familiar users from our user table let's just select star from users that's how we do

[10:14] a full table scan there in Dynamo DB what you end up doing is that you'll have some SDK in whatever your native programming language is that allows you to query dynb and so for example if it's JavaScript you'd have dynamod db. query

[10:28] and then you pass in some parameters and it returns your data and so let's look cases would be so for select star from users it's just specify as the params the table name and that's going to return to you all the items in that

[10:42] table now a little bit more complicated what if we want to add a wear Clause so we want to select star from users but only where the user ID is 101 well now our prams is still table name users but we add an additional expression and we

[10:56] ID and that ID is 101 so pretty straightforward rather intuitive easy enough let's look at one more here uh what about in the case where we want to sort right so we want to sort by some

[11:11] order date this is that sort key comes in handy and so now we can have that table name of orders we can still have some condition like we had before but we'll have this scan index forward and so uh if you put this as false it's

[11:24] descending if you put it as true it's ascending so it's a different language a directly there within your code it's rather intuitive easy enough there's actually one other thing I want to show you and that's transactions and so I've

[11:38] think I'm going to continue to do so transactions were introduced in Dynamo DB in 2018 a transaction is just a way to ensure that group of operations either all happen successfully atomically or none of them happen at all

[11:52] typically it was like if you need Atomic operations then you would have to go with the SQL database in your system design interview not anymore Dynam DB supports transactions and so see how it does it begin transaction this is in SQL

[12:06] and then we have two uh things that we want to do here this is in the case someone's balance and add $10 to someone else's balance like in a banking application and here's how we would do it in Dynam DB we would do dynamod db.

[12:19] transactions transact right and then we just have a list of the two operations that we want to have happen so in this case reduce 100 from one person and add 100 to someone else so super cool Dynam DB even supports transactions you can

[12:33] tell your interviewer that now let's talk just a little bit about what's actually happening under the hood here and so we've chatted about how Dynamo DB scales by using consistent hashing on items partition keys in order

[12:47] to determine where to physically store the item and so just as a reminder and I have a diagram here for you consistent hashing works by mapping each node in Hardware or you can also have virtual nodes but let me handwave that for a

[13:01] moment to an assigned position on a circular ring and so here you have a circular ring it goes zero to 100 at least in our contrived example and then to take some partition key you're going to Hash it and that hash is going to

[13:16] need to take a modulo so that it falls on the ring but regardless you're going it's going to give you a number in our contrived example 0 to 100 and so we're going to go to that point on the ring in this case 20 8 and then we're going to

[13:30] walk clockwise from there until we find the first node this is the node that that item is stored on and so the nice thing about this is that if you remove nodes like for example that one then the only things that are affected are the

[13:42] direct neighbors nothing needed to be redistributed over here for example and about consistent hashing they go into far more detail than that but I just wanted to do a quick reminder there and emphasize that that's what uh Dynamo DB

[13:55] is using under the hood on the partition Keys now when it comes to to fault tolerance excuse me and availability dynamodb is using asynchronous replication and so this just means that all wrs come into a single leader node

[14:11] and that leader node then asynchronously forwards that data on to replicas that are in different physical locations different availability zones they're different availability zones they're called in Dynamo DB and so Dynam DB

[14:24] interesting it uses something called saloy quorum which means that even if some network connectivity is severed like here boom that but imagine network connectivity was severed there then we temporarily

[14:39] allow rights even during those network issues and we just fix any of those consistencies later behind the scenes cool so that's what's happening there leader you asynchronously replicate now in the majority case in the default case

[14:53] we can read from either leader or any of the replicas but this just might mean consist consistency so if somebody in some part of the world just wrote some data to the leader and it had only been propagated to the replica 1 but not to

[15:05] replica 2 yet and someone else was reading data from replica 2 they might you know maybe milliseconds uh you know it depends on what your network congestion might be but that's the eventual

[15:19] consistency case and that's the default for Dynam DB but oh where was it I had a for Dynam DB but oh where was it I had a box here somewhere but bang in 2018 DB introduced support for strong consistency so now while eventual

[15:34] consistency is still the default you can enable strongly consistent reads and strongly consistent reads enabled then all of your reads are going to come off of the leader and you're not going to have any of that replication lag and so

[15:48] has some Downstream negative consequences but more importantly what is that in your non-functional requirements when you consider cap need High availability or strong consistency it used to be the case that

[16:03] if you chose strong consistency then your database Choice almost necessarily has to be SQL that's not the case anymore now you can explain in your interview that I need strong consistency which just means that I'm going to

[16:15] enable strongly consistent reads within Dynamo DB and just like we were showing configuration your AWS dashboard you're just going to check a box pretty straightforward all right at this point you have the basics down you largely

[16:30] order to introduce and use competently Dynamo DB in a system design interview Advanced features here that are supported by Dynamo DB that are really neat and handy to pull out in your interview the first one is Dax uh it's

[16:47] short for Dynam DB accelerator and what in-memory caching layer that provides microc response times for heavy read loads uh with Auto automatic read right through caching and so what you can do

[17:01] in your dashboard is you can enable uh Dynam DB accelerator Dax and this is just a caching layer that's going to sit in front of Dynamo DB automatically and now you can query this cache and you can configure uh how you want to handle your

[17:15] but the important thing to note is that if you introduce dynamodb in your interview and then you decide that you need a cash in most cases not all but in the overwhelming majority of cases you want to say that you're just going to to

[17:29] introduce reddis or something else this would be um suboptimal for you to do example of a place where you would need this it's the same place that you would had a system design interview for something like Yelp and you were

[17:44] designing a basic um you know review service for businesses you have a client you have some business service and then interfaces with Dynamo DB which has all of the business information and maybe you want to make retrieving businesses

[17:58] fast by ID I would argue you don't really need to enable Dax for this you're already at you know single- digit millisecond latency with Dynamo DB do you really need microsc latency probably not but what if there are some queries

[18:11] like a query to get all of the reviews and find the average review now of but bear with me for the example it might be that you just cache the results might be that you just cache the results of that query in Dax and in doing so now

[18:26] CU you don't have that quote unquote more expensive aggregation it's just as simple as enabling Dax and so in your interview you can say as much I have anim ODB in order to make sure things are more efficient I'm going to cach

[18:39] are more efficient I'm going to cach this data by enabling Dax cool let's one of the advanced features I want to introduce that's useful in a system design interview this one's Dynam DB streams and so if you've watched any of

[18:52] our other videos you've seen me use sometimes change data capture and what this is is it basically means that you can track any cable modifications so changes that happen in dat in Dynamo DB like inserts updates deletes Etc they

[19:06] can trigger some Downstream process this Downstream process could be like to put it into a que to have some Lambda worker do something off of it um one concrete example is here let's look at this imagine that you're designing Ticket

[19:22] Master and you have an event service which has an events DB and it has all the events that are available to users but users need to be able to search for events and this searching through events is fairly complex and expensive it has

[19:34] geospatial queries it has um you know full text search Etc and so we opted to use a search optimized database like elastic search but now we want to make sync they remain consistent at least eventually so and so one thing that we

[19:50] can do is just have every time a new event is added or updated to the events event is added or updated to the events DB we can use uh Dynamo DB streams ddb streams in order to make sure that that update is reflected in elastic search

[20:05] and so you would configure Dynamo DB streams here you would have some Lambda system design interview typically but it's an abstraction you'd have some Lambda here that takes any changes and then goes and issues some query to

[20:18] elastic search or some right operation to elastic search in order to update the data there accordingly so if you need CDC and you're using Dynamo DBS uh you can just say that you'll use ddb streams all right so let's answer the

[20:33] million dollar question that you are all here for uh when should you use Dynamo DB in an interview well the reality is for most system design interviews Dynamo DB works great So my answer is like most of the time U but in the same way

[20:46] postgress works great so if you ask me the same question about postgress I would say it also is most of the time these Technologies in databases have these Technologies in databases have converged so much uh that they all or or

[20:58] popular ones work for the majority of cases don't ever think this but the more interesting question might end up being when should you stay away from Dynam DB and I think there's three rules that you can follow here the first if you is if

[21:12] you have really complex query patterns and so this is going to be joins across a bunch of tables a lot of subqueries then it's true dynamodb might be able to pull this off uh with proper modeling but a SQL database is still probably the

[21:24] better tool for the job and then second is that if you need transactions across multiple tables Dynam DB can support this but there's a limit on the number of items that can be items again or rows right that can be in a given transaction

[21:38] I think this number is at 100 right now don't quote me on that um but this may might be something that you want to stay away from and use SQL for instead and modeling typically the sign here at least in production is if you have a ton

[21:52] of gsis or lsis Global secondary indexes or local secondary indexes then you may be abusing dyn a bit and you should consider going with SQL instead but all in all the main takeaway and I think we suggest this on the website as well if

[22:06] for your interviews I would learn about postgress and I would learn about Dynamo DB and you can largely use either of them in just about any circumstance and pros and cons appropriately in order to make the uh most effective

[22:21] decision all right so zooming out here both figuratively and literally you can ground in a fair fairly short period of time so you should have the basics under system design interview you're feeling more confident of course my advice from

[22:36] here is to go actually play with it on there are free accounts with AWS you can set up a a simple Dynamo DB instance query it have some fun get some equip you with most of what you need in order to answer interviewer's questions

[22:50] understanding of when to use it in an interview so thank you all for watching questions please leave them in the comments of course if I said something wrong let me know in the comments and as always good luck with your interviews

[23:04] always good luck with your interviews talk to you soon