---
title: 'Elasticsearch Deep Dive with Ex-Meta Senior Manager for System Design Interviews'
source: 'https://youtube.com/watch?v=PuZvF2EyfBM'
video_id: 'PuZvF2EyfBM'
date: 2026-08-04
duration_sec: 2643
---

# Elasticsearch Deep Dive with Ex-Meta Senior Manager for System Design Interviews

> Source: [Elasticsearch Deep Dive with Ex-Meta Senior Manager for System Design Interviews](https://youtube.com/watch?v=PuZvF2EyfBM)

## Summary

This video is a comprehensive deep dive into Elasticsearch, presented by Stefan, a co-founder of Hello Interview and former senior manager at Meta and Amazon. It is designed to help viewers understand Elasticsearch both as a tool for system design interviews and as a distributed system with complex internals. The session is split into two parts: first, how to use Elasticsearch in a system design context, and second, an exploration of its underlying architecture, including Apache Lucene, shards, and query optimization.

### Key Points

- **Introduction to Deep Dive Series** [00:02] — Stefan introduces the deep dive series, aimed at providing interview-relevant knowledge on technologies like Elasticsearch. He shares his background as a senior manager at Meta and Amazon with over 2,000 interviews conducted.
- **Session Structure** [00:45] — The session is divided into two parts: first, how to use Elasticsearch in system design interviews, and second, peeling back the layers to understand its internals and design decisions.
- **Search Experience Components** [02:45] — A typical search experience includes criteria, sorting, and facets. Using a bookstore example, he explains how users search for books under $13, sort by publish date, and refine by publisher.
- **Key Concepts: Documents and Indexes** [04:26] — Documents are JSON blobs, and an index is a collection of documents. Within an index, a mapping defines the schema of searchable fields and their types, such as price as a float.
- **Creating an Index via REST API** [05:51] — Elasticsearch provides a RESTful API. To create an index, you make a PUT request to /{index}. You can optionally specify a mapping to avoid type inference errors and reduce overhead.
- **Field Types: Text vs Keyword** [07:12] — Text fields support full-text search (e.g., searching for 'great' in 'Great Gatsby'), while keyword fields are opaque and suitable for exact matches like categories. Keyword lookups are fast but don't support partial matches.
- **Nested Documents and Denormalization** [07:58] — Nesting reviews within a book document allows answering queries with fewer requests, but it's costly if reviews change frequently. Denormalization is key for Elasticsearch performance.
- **Adding Documents** [08:40] — To add a document, POST to /{index}/_doc with the document body. Elasticsearch responds with the document ID, index, and version number, which is crucial for handling concurrency.
- **Updating Documents and Versioning** [09:35] — Updates can be done via PUT with the full document, but this can lead to last-write-wins conflicts. Using the version number in updates allows rejection of stale updates, enabling client-side conflict resolution.
- **Field-by-Field Updates** [10:33] — POST to /{index}/_update allows partial updates, e.g., changing only the price. This reduces the risk of overwriting other fields.
- **Search API Basics** [11:01] — The _search endpoint accepts JSON queries. For example, a match query for 'great' in the title, combined with a range filter for price less than $15.
- **Search Response Structure** [12:20] — The response includes metadata like time taken, timeout status, number of shards hit, and the hits themselves. The _source field contains the original document, and you can request specific fields.
- **Sorting Results** [13:20] — Elasticsearch allows sorting by fields like price and publish date. You can also use scripts (Painless) for custom sort orders, and sort by nested fields like the highest review.
- **Relevance Scoring with TF-IDF** [14:37] — The default scoring algorithm is a variation of TF-IDF, which boosts documents where the search terms appear frequently and are rare across the corpus. This helps rank relevant results.
- **Pagination: From/Size vs Search After** [16:50] — From/Size is simple but inefficient for deep pagination. Search After is more efficient as it restricts the search space based on the last seen values, but both can miss documents if the index changes.
- **Point-in-Time Search** [19:35] — Point-in-time search creates a snapshot of the index, ensuring consistent pagination. It requires a keep-alive and should be deleted after use to free memory.
- **Using Elasticsearch in System Design** [20:31] — Elasticsearch is ideal for complex search scenarios like geospatial, vector, or full-text search. However, it's not a primary database due to weaker durability and availability guarantees.
- **Elasticsearch as Secondary Store** [21:10] — Common pattern: use a primary database (e.g., Postgres) and use change data capture (CDC) to sync updates to Elasticsearch. This ensures eventual consistency but allows read-heavy workloads.
- **Denormalization and Avoiding Joins** [22:30] — Elasticsearch doesn't support joins. Flatten data to avoid joins, and consider if you even need Elasticsearch for simple search needs—sometimes a database's built-in search suffices.
- **Elasticsearch Internals: Lucene and Nodes** [23:27] — Elasticsearch is an orchestration layer over Apache Lucene. It handles distributed aspects like nodes, while Lucene manages data organization and speed on a single node.
- **Node Types in a Cluster** [25:06] — Master nodes handle administrative decisions, coordinating nodes act as the API layer, data nodes store indexes, ingest nodes process documents, and ML nodes handle machine learning tasks.
- **Shards and Replicas** [28:06] — Indexes are divided into shards, each containing a Lucene index. Replicas are copies of shards that improve read throughput and availability. Shards have a limit of 2 billion documents.
- **Lucene Segments and Immutability** [29:41] — Lucene stores data in immutable segments. Updates are soft deletes: the old document is marked deleted and a new segment is created. This immutability enables caching and concurrency benefits.
- **Inverted Index** [32:46] — The inverted index maps terms to document IDs, enabling fast lookups. It's a key data structure for search, but requires tokenization and stemming to handle variations.
- **Columnar Store: Doc Values** [35:31] — Doc values store fields in a columnar format, allowing efficient sorting and filtering on specific fields without loading entire rows. This complements the inverted index.
- **Query Planning and Optimization** [38:01] — Coordinating nodes optimize queries by choosing the best execution plan based on statistics, such as which terms are rarer, to minimize data transfer and processing.
- **Ingestion Pipeline** [41:07] — Documents are sent to ingest nodes, processed, and forwarded to data nodes. Data nodes index into Lucene segments and acknowledge back to the client.
- **Search Execution Flow** [42:05] — Clients contact coordinating nodes, which plan and distribute queries to data nodes. Data nodes query Lucene indexes in parallel, using inverted index and doc values, then return partial results to the coordinating node for merging.

### Conclusion

Elasticsearch is a powerful search engine that excels in complex, read-heavy search scenarios, but it's not a primary database. Understanding its internals—Lucene, shards, and query optimization—can inform both system design decisions and interview responses.

## Transcript

section of our deep dive Series today we're going to be talking about elastic search which is one of the most popular and Powerful open-source distributed search engines reminder that our Deep dive series is intended to give you
your interviews and so what we're going to be trying to cover here are both how Technologies can be used in an interview and an assistant design setting as well understand the internals and answer questions
uh that may come up I'm Stefan I am one of the co-founders of hello interview I've worked as a senior manager at both meta and Amazon and over the course of my career conducted north of 2,000 interviews very familiar with how
process and hoping to share some of those insights along the way this session is going to be divided into two parts in the first part we're going to talk about how to use elastic search this is particularly relevant for
interviews where search is actually not the primary functionality or in certain types of interview like the product design interview at meta or interviews for full stack engineers at startups where your interviewer wants to see you
solve problems not necessarily you have deep knowledge of how distributed systems work in the second half of the session what we're going to do is peel back the layers of elastic search and talk about some of the design decisions
function the goal there is both to give you some insights into how it actually functions but also some inspiration for your own designs especially as they pertain to more infra style interviews before we
get too deep just a reminder that all of this information is available on our website hello interview.com there we've collected a bunch of deep Dives on technologies that we really think you should know just like elastic search as
well as common problems that you might encounter in a system design interview walk through some of the different choices that you might need to make and then giving you an indication as to which ones are great and which ones are
probably not the ones that you might want to use in an interview we've also recently launched our guided practice product guided practice is solving this problem that a lot of people watch videos they read blogs and they kind of
fail to integrate the information that they've consumed with guided practice you're actually going to work through a problem you'll submit your answers on a whiteboard word by narrating your response and then you'll get
personalized feedback on how you did what things went well what didn't go well and how to improve this is a great way to make sure that you're actually retaining the information that you're taking in and there actually is a guided
practice for post search which is a perfect complement to this video in the first half of this video we're going to talk about how to use elastic search in a system design setting for that you first need to talk about search so what
is search well typical search experiences are comprised of a few different pieces first I'm going to give the search some criteria let's pretend actually going to use a bookstore as a motivating example throughout this video
so maybe I want to search my bookstore for books that have the title that less than $13 that's potentially describing a lot of books so I might want to sort those books in my search results in that case
most recently published books that contain the word great in my search result page I might also have an option to break down the search and refine what I've described these are typically called facets or faceted search and one
example is I might want a refine by the publisher maybe hash it or Simon and Schuster boo hash it for suing the internet archive anyway the net of a search experience is a set of results basically a set of documents in this
basically a set of documents in this casee the documents represent books that I want to return but they could really be anything we can think of this as arbitrary Json blobs that we might wearn in return while elastic search does
support other use cases like analytics use cases as part of the elk stack I'm not going to talk about this too intensively in this session just because search is really rich and I want to go into more detail now how does elastic
search think about search well there are some key Concepts that we have to identify and name before we can go into more detail about that the first and most important concept is the document like I mentioned documents aren't
necessarily PDFs or things like that they're just Json blobs and they can potentially contain anything the next primitive that we're working with in elastic surch is called an index and index is kind of an overloaded term in
computer science index is usually referring to how we structure data to make retrieval really fast in this case an index is just a collection of documents that we want to make searchable within an index we have two
relevant Concepts one is called a mapping and one is called a field the mapping is going to specify in some sense the schema that I want to be searchable on my documents so my documents might have dozens of fields
but my users may not need to search all of those my mapping in some sense is telling the search engine elastic search which of those fields are pertinent and then what are their types so in this case I can tell elastic search that the
price is always going to be a floating Point number and as a result elastic search can organize that information so that I can have range queries and I can find Price Less Than $13 so with our terms in place we can
now talk about how we actually bring docs into elastic search and how we make them searchable elastic search provides a restful API and it actually actually has a pretty good one so if you haven't designed a restful API before it's
actually a useful one to look at both so you can get some ideas about what a solid production level API looks like and how they handle some complicated Corner cases the very first call that we're going to make is this put to slash
whatever our index name this is creating a books index and I can specify some configuration parameters but all of them are optional I'll get into what shards and replicas are a little bit later important thing here is this creates me
an index I don't have to specify a mapping but I can and the reason that I might want to is because if elastic search can't infer the types of the
fields which ones I want to search particularly complex configurations it may make some mistakes there's also a lot of overhead to having elastic search guess at what your mapping might look like because it's going to be indexing a
bunch of fields that you may not care about now note the field types here have a ton of implications we talked about how the price we wanted to be able to do range search on so that's available here note also that we've got these text and
keyword types text allow us to do rich full text search so if I want to search for Great in Great Gatsby I can do that in a text field a keyword on the other hand is opaque each of the tokens is kind of a unit and so this is really
appropriate for places where for instance my set of categories is fixed maybe in the UI and the users can click or select from the available categories lookups Lightning Fast think a hash lookup but it won't allow me to for
instance search for all categories that contain the word fiction when that might include non-fiction note also here that I am including this nested reviews this
if you're famili with normalization and denormalization might be a little bit clear but what we'd like to be able to do is answer our search queries with as few requests as possible and by nesting the reviews in the document we can
basically construct queries where we can say I want specific types of books that have certain reviews now this is not without its downsides if reviews are without its downsides if reviews are changing a lot or if my queries rarely
really EXP expensive for me to maintain that way and I might actually be better served by creating a separate index for them but in this case I'm showing you what this might look like and assuming that reviews aren't going to be shifting
a lot but that customers care about them next let's talk about adding documents next let's talk about adding documents to our Index this is dead simple I make a post request to this undor doc endpoint inside my index and I pass all
of the documents contents when I do this elastics search is going to respond with a basically success message that tells me a number of details for instance what
was this document's identifier what was the index it was created into and this version number the important thing here is that elastic search is a distributed system we could be making updates from any node and they might take some time
to percolate through the system so it's really important that we handle collisions and concurrence this is particularly relevant when we this is particularly relevant when we start to talk about updates for updates
the simplest possible API is to make a put request to this doc endpoint with the ID that I'd seen earlier I pass in the entire document body and I can update it now this is probably Overkill if I just want to change the rating on
one of the reviews for instance but it also has some downsides in that if two updates are submitted simultaneously the last one wins this is where that version identifier comes in what I can do is in my update I can pass in the version
number and what elastic search will do is it will reject that update if that version number doesn't match this gives me a chance to handle the Collision or the conflict on the client my client can then go in query what the latest version
is take a diff decide what to do and resubmit it to request this is actually a pretty common pattern in distributed systems and it's just a way to ensure that we would anticipate another strategy here which actually doesn't
solve the raise conditions but solves part of the problem is that I can make field Byfield updates by post posting to this underscore update endpoint and in this case I'm simply bumping the price
from $33.99 to $4.99 Okay so we've talked a lot about how to insert and update documents as if elastic search is a document database and it kind of is but the key thing here that we want to get into is search well fortunately
search is really straightforward too there is this nice API endpoint underscore search which sits on the index and allows you to search using this query language that's basically Json and very SQL like in nature so if
for instance we wanted to allow searches by title where it contains a specific word then we can just simply do a match query with great in the body if we wanted to extend that by saying that we also want it to be less than $15 we can
do that and then if we wanted to have a query of books where the reviews had this comment excellent and their rating is greater than or equal to four out of five we can do that too now note these are get end points you can't actually
put this in the body of a get request it doesn't actually meet standards or you can actually stuff this into a query parameter but you're limited by the size of a URL which I think is 2048 someone in the comments please correct me it's
also possible to post here although that's a little bit out of Canon in that search requests you're expecting them to get they're not actually mutating anything but the important thing here is this API is really nice and convenient
and allows you to specify arbitrary search criteria the first part of the search product that we talked about earlier now once I've got search results API that looks like this elastic search is going to conveniently report to me
how long that search took whether it timed out how many shards it hit again later and then it's going to tell me how many hits and it's going to return the contents of those documents to me now this is configurable and in some sense
you may not want to return the entire document in your search results but elastic search has this underscore source which is basically that Source document that you originally uploaded it's also possible to return only
specific fields from the document I may only want the ID or the title I'm always going to get the document ID inside of elastic search there's also this field called score which is trying to calculate the relevance of the document
to the query that I've input and that brings us to the second part of our search product which is the Sorting just like searching and filtering elastic search has a pretty extensive API to allow you to specify whatever sort order
you want so in this first instance I'm sorting exclusively by Price whereas in the second instance I'm sorting first by price and then by publish date remember that I need one field to be the first and most important sort order the
remaining fields are basically TI Breakers I can also specify scripts elastic search has a language they call pain L which allows me to specify basically formulas that I can use to calculate so this would be the price
with a 10% discount doesn't make any sense as a sort order to be frank because the sort doesn't actually change but you get the general idea that you can pass in a formula here I can also sort by those nested objects and Fields
so if I wanted to sort by the highest review I could do that as well elastic search will allow me to specify the sort order very rigidly if I want to I can tell it exactly which fields to order on or I can use that underscore score field
that we talked about in the result set earlier which basically orders by relevance now relevance is a really deep topic there's some machine learning and statistics aspects to this but some of the simplest algorithms are actually
pretty intuitive and easy to understand so the most simplest of them all is one so the most simplest of them all is one called tfidf term frequency inverse document frequency and there's two fundamental observations here first the
most relevant document probably contains my term more frequently if I'm searching for elastic search I want it to mention elastic search a lot the second thing is I'm going to favor those words that don't appear very frequently across all
my documents so as an example if I'm searching for elastic search guide and document which is functional and elastic search guide and then the second is a search guide and then the second is a summary of a court case about some
weapon being handed to a guide and the prosecutor using the technology which one should I prefer they both contain elastic search and guide well the first term frequency is going to tell me that elastic search appears very commonly in
this document so it's going to get a score boost just by the number of times that elastic search appears the next thing we're going to note is that guide actually appears in a lot of documents it not only appears in my elastic search
guide it appears in court cases it might appear in a bird watching guide and as a result it's a relatively uninformative term so what tfidf is going to do is it's going to score those items which contain the keywords that are densest
and don't appear in many other places higher than those keywords that maybe don't appear very often or appear in almost all the rest of the documents in my Corpus the default scoring algorithm is not actually tfidf but it's a close
approximation you can assume that elastic search is basically making some statistical inferences based on your documents if you don't specify a specific sort order now that we filtered and sorted the results we're almost
ready to send them to the user but we've got one last step which is we might have potentially return and most people aren't going to scroll to actually see all of them it be madly inefficient so we need to Institute some sort of
pagination there are two types of pagination stateful and stateless stateful pagination requires that the server keep track of something so that way it knows what the next page is usually you're talking about cursors
here in stateless pagination we're going to pass in some sort of parameter that allows us to localize approximately where in a result set we'd like to look so as an example this from and size query is simply going to take results
query is simply going to take results starting with the index of from zero and pages of size 10 it's going to take the first 10 results and return them to me when I query again for the next page I'm going to increment that from to 10 I'm
going to keep the size the same and that will give me the next 10 for a total of 20 very straightforward I can also use this search after so in this case I'm this search after so in this case I'm sorting by date and ID when I get to the
sorting by date and ID when I get to the bottom of my results I am going to have a date after which there are no more results in subsequent pages and an ID after which there's no more results what
this is is telling elastic surch is it saying don't return me any results that saying don't return me any results that might be before these two quantities these are both comparable but search after is preferable particularly in the
cases is where you want to go deep into the pagination because this requires that me get all the results and then we're going to slice into that result set based on the inputs whereas this actually allows me to restrict the
documents that I'm querying based on this search after after I've gone deep into the result set I may actually have a much smaller Corpus of things that I can search over and so elastic search has to do less work it's a kind of
clever idea now both of those those search types have a problem which is the index is constantly changing new documents are being added some are being
deleted in the worst case scenario you actually have a document which would have been on the next page but changed maybe due to a deletion and bumped up to the page that you were just on if that's the case then your pagination whether
you choose search after or the from and size is going to miss a document so how do we deal with this elastic search supports a point in time search and the
supports a point in time search and the idea here is I can create a point in time where elastic search is going to snapshot what the index looks like at that period this is going to return me an ID in this case
460 and I can use that point in time while I'm searching note I need to specify a keep alive this is actually pretty expensive to maintain and so I need a signal to the server that I still want it for one more minute every time I
want it for one more minute every time I search but what this does is it searches based on that snapshot and will guarantee that each page of results is consistent when I'm done I need to delete that point in time or let it
expire so that I'm not using excess memory on the server but this eliminates at least a small class of problems that you might have with respect to pagination how this point in time is implemented is an interesting point of
discussion for the next section of our Briefly summarize how you can use elastic search in your system design interview typically elastic search is
going to be for complex search scenarios if you need to do geospatial indexes if you need to do Vector search if you need to do simple full Tech search is probably your friend it
doesn't necess NE necessarily mean that elastic search applies in every scenario that you need to be aware of first of all elastic search is probably not going to be your primary database while the
API that I described looks a lot like a typical document database elastic search's durability and availability guarantees aren't actually that good and over time most people have relied on setting up a primary data source be it
post grass or Dynamo DB and then using change data capture CDC to basically push updates from that database into elastic search the idea here is elastic
search is going to be a little bit behind your database but it will have elastic search is going to work best with read heavy workloads if you've got a lot of Rights happening or if your
rights dominate your reads you may want to consider whether elastic search is you can change the nature of your workload maybe limit the number of Rights or batch them out but generally speaking elastic search is going to
perform best when we're reading a lot of data if you're using elastic search you have to be able to tolerate eventual consistency it's going to have a delayed version of the data eventual consistency is basically endemic to the design of
elastic search if you need strong consistency if you need to check for lines you're going to need to use a different database but if you're not using it as your primary database you already know that denormalization is
really key to making elastic search functional if you've got a particular search page make sure that you flatten all of your data and that you're not trying to make nested relational queries you basically shouldn't need to do joins
in an elastic search search if you do you're going to be in trouble first of all because joins aren't supported and secondly the way that you might bolt on joins by doing lookups outside of elastic search is really going to
thing to consider before you use elastic search is whether you actually need it elastic search clusters are not a trivial matter to operate and maintain it's a new technology in a lot of cases a simple search capability in your
existing database might cover your needs if you need full teex search if you need to index billions of documents elastic search is your best friend but if you've got simplistic needs aim for Simple Solutions you don't necessarily need to
dig in so far we've talked about elastic search primarily from the angle of a user or a client and from an interview perspective this is still incredibly useful in a lot of interviews what you need to be able to do is to point out
how it might fit into your overall design if you're designing Twitter search might be a peripheral concern for your overall design and delegating that
to elastic search is a completely sensible design choice in other interviews either because your interviewer explicitly tells you don't use elastic surch I'd love to see how you implement some of the Core Concepts
or because they tell you to implement a system which seems a lot like elastic search being able to understand what goes on beneath the covers of elastic search is going to be helpful and that's really what the second section is about
understand about elastic search is that it is in some sense an orchestration layer for Apache Lucine which is a low level search functionality elastic search handles the distributed systems aspects the API how
distributed systems aspects the API how to manage nodes how to manage backups how to keep availability and replication whereas Lucine is trying to make sure that the data is organized and that it's fast Lucine operates on a single node
elastic search operates at a cluster level and so as we're peeling back the be progressively talking about the different layers inside of elastic fundamentals of Lucine the first thing to understand about elastic search as we
kind of peel back the layers of the onion is the various nodes that make up a cluster now an elastic search cluster can have a single host or server node is can have a single host or server node is not necessarily a physical machine and a
physical machine can take on the responsibilities of multiple nodes but at a conceptual level these are the respon responsibilities that exist inside the elastic search cluster that need to be performed by some Hardware
somewhere the first not type is our Master node or our Master eligible node and what happens here is when we spin up our cluster we are actually specifying seed nodes or seed hosts which elastic
search will try to connect to and then we'll elect a master of the cluster we only want one master node so it can make administrative decisions like when we create indexes or which nodes are actually part of the cluster
but the important thing about master eligible nodes is that there is only one master that is elected at any point in time the next node type is the coordinating node and you can think of this as kind of the API layer to the
cluster the vast majority of requests that are coming into my elastic search and what cordinating nodes are doing are they're taking those requests parsing them and then passing them to the relevant nodes that have the data that
we need in order to return the search results users are for the most part interacting with these coordinating nodes data nodes are where our indexes are stored they are functionally where we keep our documents and they're
probably the most fundamental of all the node types we're going to have a lot of them data nodes need to have really heavy IO either lots of memory or plenty of fast dis because the coordinating nodes are going to be sending search
requests and they expect the data nodes to be able to respond the ingest nodes are like the backend for our service it's how the documents make their way into the cluster and finally the machine learning nodes are not always used but
are for machine learning intensive tasks and the idea with the machine learning node is these nodes might require access to a GPU and are going to have very different access patterns now each of these Noe types by virtue of their
responsibilities actually imply different Hardware my master nodes need to be incredibly robust and reliable it's very important to me that they're not going down my coordinating nodes are going to do a lot of network with
network traffic with the outside world my data nodes as I mentioned are actually going to have a lot of uh dis IO or need lots of memory and my ingest analysis of documents they typically are more CPU bound and so what this implies
is when I set up a cluster I might actually choose different server or VM type to take on these node responsibilities in order to optimize my cluster remember that the fundamental grouping of elastic search documents is
grouping of elastic search documents is an index this might be my books index or my reviews index inside of an index I actually have shards and replicas and these shards are actually going to be responsible for both containing all of
my documents but also building up index data structures to make searching fast the shards are mutually exclusive if I've got a document it will be assigned to one Shard or another and the replicas are basically going to be carbon copies
of those shards I might have for instance a replica of a Shard across multiple machines and what this allows me to do is when I'm searching my coordinating node can choose whether it wants to use Shard 1B or Shard 1 a and
spread the load across them so by replicating these shards I actually gain throughput particularly on the read Side by sharding I can spread out very large indexes across many different machines I believe A Shard has a hard limit of two
billion documents which is a whole lot but even if I've got a smaller number than that I still may want to divide those documents either because the aggregate size is too large for a single machine or because I want to spread the
load across my cluster to make my queries faster this this is the central idea of elastic search is that these shards are breaking down my data and inside of each Shard is this Lucine index the Lucine indexes are one to one
with elastic search shards basically A Shard is just encapsulating a Lucine index inside Lucine index we have segments and the way that Lucine works
is basically by accumulating these immutable segments what happens when I create a new document is Lucine will first try to batch that document into as
many rights as it can contain before flushing it out to create a segment with multiple documents when I add more documents later I might create a new segment and then at some point in the future Lucine may decide that it will
merge these segments into a third segment and then delete the old ones the important thing is that these segments are immutable they're not changing now are immutable they're not changing now we support updates in elastic search how
does that actually work from a Lucine perspective well with updates what we're perspective well with updates what we're doing is a soft delete so Lucine inside these segments maintains a map of deleted documents so if segment two
contain document one two and three and I deleted document two this segment will actually store that ident ifier two internally and then when I query that segment I will remove any results that contain two it's been deleted when I do
an update what I'm going to do is soft delete I'm going to mark that document as deleted in the old segment and then create the updated document in a new segment this maintains this immutability aside from that map of deleted documents
these segments aren't changing and it's one of lucin's more powerful features one of lucin's more powerful features because it enables us to exploit a bunch of caching and concurrency benefits since segments aren't necessarily
changing when I load them from memory I can do that in a compressed format and I guarantee that they aren't changing underneath me I also can have multiple
readers touching the data in the segment in a concurrent way without worrying about race conditions or contamination the segment data won't change it's readon in some respects and so this Lucine idea of maintaining these
Lucine idea of maintaining these immutable segments is really key to both the performance of elastic search it allows it to be very fast but it also puts some limitations that we discussed earlier on being more read heavy and not
being able to deal with updates or insertions as quickly as maybe other systems that might allow mutable data inside the Lucine segments we need to store all of our documents conceptually you can think of a segment like
containing documents but the important thing is that those searches need to be fast it's not enough to just dumbly store our data in these segments we need to make sure that we can search quickly Lucine employs several techniques in
Lucine employs several techniques in order to pull this off one of them is a reverse or inverted index let's talk about that briefly so when you want to make things fast with computers you've kind of got two
options first you can consider how you arrange the data in memory if you store an array randomly you may need to scan over the entire array in order to get
the thing that you're looking for if you sort it you can find an element in log and time if you know exactly what you're looking for if you want to do a lookup you can do that in O of one time with a hash table so changing how you organize
the data is one key the next key is being able to use copies if we can make a copy of our data and store it in a slightly different way then we can potentially make access to all of the data really fast now of course we trade
off storage there's a bunch of costs that are associated with it but those are really the two options for how we can make systems fast so the very first data structure this inverted index is basically making both those trade-offs
what we're doing here is we've got a set of documents the first one's says Lazy Dog it's ID number 12 and then the second is ID 53 and it says lazy days of the best day in my index what I'm going to do is store a dictionary of the words
that are in the documents to the identifiers of each document that contain them and what this will allow me to do is I can very quickly in this to do is I can very quickly in this index look at all documents or all yeah
I can look at all documents that contain the word lazy I simply go to the lazy uh index index I can do that with a hash lookup and then I can grab all of the documents that it contains now there's a bunch of complexity that comes into
these indexes for instance I might have variants of lazy that I want to be able variants of lazy that I want to be able to include I might have other ways of spelling dog that I want to collapse together and certain words like R may
appear in almost all documents and so maybe we don't want to include them this is fundamentally a question of tokenization or stemming or litiz and we
won't go into too much detail about this but this is an important consideration in order to make these indexes quick now my inverted index gives me a series of my inverted index gives me a series of document IDs but my search query might
actually have a sort order maybe I want to sort by Price how can I actually make sure that my results match that query the next step for us is to think about how the data is actually arranged in memory there's fundamentally two
different ways that you can think about how to organize a table of information like what we have here the first is what you would do in a SQL database each of these rows would be stored basically contiguously and you would have a bunch
contiguously and you would have a bunch of structs or or fields that are going of structs or or fields that are going to represent each of these inputs so as an example this is a piece this is a piece and this is a
piece this is a piece and this is a piece my table is an array of these rows now this makes it really easy for me to get the entirety of a row given an identifier so if I know what my ID is and I can look up the the row uh index
then I can very quickly pull all of the data for that record and in SQL databases this is a very common operation but what we want to do in this case with our Lucine segment is we want to take a bunch of Doc IDs and resolve
one of the fields we for instance want just the price we may not even care about the title and the author in our query and so in this case what we want query and so in this case what we want to consider is a columnar store in this
case we're actually going to store the data as a bunch of columns that we then concatenate together so instead of thinking of the table as rows that are sitting on top of each other we'll think of it like fixed length columns and what
this allows us to do is it allows us to get a single or a small number of field values across all of the documents very quickly if I have a large number of Doc
quickly if I have a large number of Doc IDs I can grab all of their prices without iterating and crawling over the entire table grabbing each row even though I don't care about title and author in Lucine this is an index called
doc values and Doc values contains these columnar fields that allow it to do very columnar fields that allow it to do very quick sorting and filtering based on those doc IDs that are identified by the inverted index now while Lucine has some
very low-level optimizations for how data is organized to make the queries as fast as possible there are also other optimization opportunities that exist inside of elastic Sur one example of this happens in the coordination nodes
when a query is received by a coordination node it is first parsed and then there's an opportunity to optimize that query what happens is the system will basically plan what data it needs to access in order to get the results
to access in order to get the results the user wants and try to optimize that plan in order to make it as efficient as possible now what might be efficient or inefficient let's let's pretend that we've got a query that has two keywords
we've got a query that has two keywords Bill and NY and in our index we have 1 million documents that contain the word Bill and 1,000 documents that contain the word NY there's a lot of different ways that we could execute this well
first of all what we could do is we could grab all of the documents that match Bill all of the documents that that match nigh Union them together and then do a string search through all of their contents this is clearly very slow
and it would give us the correct answer of all the documents that contain Bill and NY together but it would require that we transfer the entire content of the document across the network which is just a non-starter another option is for
just a non-starter another option is for us to use these IDs so we could go in the ID for Bill and we could intersect that with Bill and we could intersect that with all of the IDS for NY now this number is
necessarily less than a thousand so a lot of these IDs aren't going to match but the IDS that have both Bill and nigh are going to be matches to our query that sounds pretty good but we can do even better than this for instance if we
start with n we don't have to transfer a million document IDs over the network we document IDs over the network and we might be able to do even better by going finding all of the documents that contain n and then searching through
their Contex to see if any of them contain bill this is a much bigger number and if the contents are small enough we can simply do an aggressive search across a very small number of documents so this is the essence of
query planning that there are a lot of different options that give the same result but have very different performance characteristic that we get to choose from when we try to execute a query and on the coordination nodes we
have this option to generate query plans and then using St statistics that we have on our index like how many documents match a particular keyword we can then go and decide which shards we want to access which Lucine segments we
need to go in query and how we can get the results back to our users as fast as possible all right so we've gone super deep to understand how Lucine works at a low level to answer search queries as well as optimizations that can happen on
coordinating nodes to do query planning and optimize the ex ution plans let's take a step back to walk through the entire layer of the onion to talk about how documents are ingested into an elastic search cluster and how they're
made searchable so the very first step is to create a document in a typical setting a client is going to send a document to an ingest node which is going to run an ingestion Pipeline and process that document before passing it
process that document before passing it along to a data node once that data node has received and index the node it's going to send an acknowledgement back to the injust node which can be confirmed back to the client so they know hey my
document's been received on that data node we are going to add that document to a Lucan index in most cases that's going to result in the creation of a new segment if we've been adding a lot of other documents there may be a buffer
and a segment might be created with multiple documents now now when it comes time to search our clients are going to contact the coordinating node the coordinating nodes are going to parse the query that
the user has submitted they're going to make a plan for how they can access the data basically which shards it resides on and then they're going to submit those to the relevant data nodes the data nodes are then going to
nodes the data nodes are then going to pass that query to their Lucine indexes which are going to run in parallel across the relevant set segments and across the relevant set segments and utilize features like the inverted index
utilize features like the inverted index and the doc values to both filter and sort the results before returning it back up the stack the data nodes are then going to return those partial results to the coordinating node which
merges those results and returns them back to the user so we've got parallelism at multiple levels you could potentially have multiple shards s that are being queried we could potentially be sending requests to different
replicas and inside the index we can parallelize the operations across multiple CPUs thank you for sticking with me through this video we've covered a ton of ground if you've got a highlevel interview coming up either
full stack or product architecture I hope you can use elastic search in those instances where you need high performance search for complex scenarios that really require immense scale if you've got an infrastructure style
interview coming up I'm hoping that learning about elastic searches internals and how Lucine works is informative and build some inspiration for some future designs that you might have I'm happy to answer any questions
that you all might have in the comments and would love uh any reactions or ways that we can improve this um in general we put out lots of great content on this channel so subscribe to us if you're interested in more system design or more
interview specific content and until next time I'm Stefan talk to you later