Docker Compose Tutorial

[00:00] In this video, you're going to learn about Docker Compose, so by the end of this video, you'll be able to use Docker Command with confidence. Hi, I'm Marsh Hamidani, and I've taught millions of people how to code through this channel and my online tools taught with Marsh.com.

[00:15] This video is part of my ultimate Docker course, so once you finish this video, if you want to learn more, then you want to look at the complete course. Now let's jump in and get started. Welcome back to another section of the Ultimate Docker course.

[00:32] In this section, we're going to talk about running multi-container applications. So I'm going to give you a real-world application with three building blocks. A front-end built with React, a back-end built with Node, and a MongoDB database.

[00:45] Once again, you don't need to be familiar or use any of these tools. Our focus here is on Docker and not on development tools. I think this is the most exciting part of this course where you can see everything coming together.

[00:57] We'll talk about Docker Compose for building and running multi-container applications. We'll also talk about Docker networking, database migration, and running automated tests. So, let's jump in and get started.

[01:09] In this section, we're going to use a tool called Docker Compose, which is built on top

[01:24] of Docker Engine. It makes it incredibly easy to start applications with multiple containers. So, Google Docker Compose install. You'll find this page, docs.docker.com slash compose slash install.

[01:37] So, at this page you can see the installation instructions. Now at the time of recording this, Docker Compose is shipped with Docker Desktop for Mac and Windows. So if you have Mac or Windows, you don't have to do anything extra.

[01:50] You already have Docker Compose. To verify it, just go to the terminal window and type Docker Compose dash dash version. So I'm running Docker Compose version 1.28.5.

[02:03] Make sure your version is the same or newer. If you're using an older version, again, Google, upgrade Docker Compose. Or you might just install the latest version of Docker. Now, back to this page.

[02:16] If you're using Windows Server or Linux, there are specific instructions you have to follow to install Docker Compose. So go ahead and install Docker Compose, and I will see you in the next level. Before we get started, I want to show you a couple of techniques for cleaning up our

[02:36] workspace. So on this machine, we have a bunch of images and some run containers. They're getting in the way. I want to get rid of them all. How do we do this? Well, you know that we can remove images using Docker image remove command.

[02:50] And here we can type one or more image IDs. Now, how can we get all image IDs and pass them here? Let me show you a cool trick. So, we can run docker image ls. We see all the images, right?

[03:03] But if you pass dash q at the end, we only get image IDs. Now we can pass this as an argument to docker image remove. So, docker image remove. Now, here we add the dollar sign and in parentheses, we type that other command.

[03:19] So, docker image ls-q. If we run this, we're going to get an error because some of these images are already running containers or stopped containers. So, we should always remove containers first.

[03:32] We're going to do that using the same technique. So, I'm going to replace image with container. So, we get all container IDs and then we're going to remove them all in one go.

[03:45] Also, I would like to add dash a here as well. This will bring stopped containers as well. Okay? We can also combine switches. That's another technique. Let's go ahead.

[03:57] Alright, we get another thing. You cannot remove a running container because I forgot to pass the fourth option. So, let's bring this up one more time. When removing, we're going to use dash F.

[04:11] Okay? Great. So, all these containers are removed. Now, let's remove the images. So, docker image ls and docker image remove.

[04:26] Great! Now, take a look. We don't have any images here and no containers, including stopped containers. So we have a clean workspace. That's one way. There is a shortcut for this as well.

[04:39] If you're on Mac, you can find the Docker icon on the top status bar. If you're on Windows, you'll find it in your notification tray. Let's click on this and then go to Preferences.

[04:51] Now on this page, let's click on the troubleshoot icon. On this page, we have a bunch of useful utilities. For example, we can restart Docker Desktop. We can also clean and purge data. This will essentially remove everything in Docker.

[05:05] Your images, your containers, your volumes and so on. Now, be aware that if you click on this, this is going to restart Docker Engine. So on the top look, you can see this animation showing that the Docker engine is not started yet.

[05:20] So at this point, if you go to the terminal window and execute any of Docker commands, you're going to get an error. So you'll have to wait about half a minute for the Docker engine to start. That's another way. So now that we have a clean workspace, next we're going to talk about our application.

[05:35] So in this section we're going to look at a real-world application with multiple building blocks. A front-end, a back-end, and a database.

[05:48] So below this video I've attached to the Z-File. Go ahead and download it. Inside that Z-File you're going to find this folder structure. We have this backend folder which is our node project. This is a basic node project that starts a web server on port 3001.

[06:04] Once again, you don't need to know no to go through this section. Then we have the frontend project, which is a React application that talks to the backend. Now, if you want to run this application outside of Docker, there are a number of steps we have to follow.

[06:18] Let's say we just check this out from a GitHub repository. First, we have to go to our backend project, install all dependencies, and then start the web server. Now at the same time, we have to open up another terminal window and do the same steps with

[06:33] our frontend project. So we have to go to the frontend project, install all the dependencies, and then start the web server. And of course, we need two more terminal windows for running our frontend and backend tests.

[06:46] And not to mention that, we should also download and install MongoDB on this machine. There are so many steps we have to follow the moment we check out this source code from our GitHub repository. Now with Docker, we don't have to do any of these things.

[06:59] All we have to do is run a single command. Let me show you. So, I'm going to get outside of the front end folder. Now we are in the root of this project. If you look, here we have a file called Docker Compose, which is used for composing a multi-container application.

[07:15] We're going to talk about that in detail soon. Now, once we have this file in our project, we can simply run Docker Compose up. That's all we have to do. Now Docker is automatically downloading this particular version of MongoDB, so it's downloading all these layers.

[07:33] Then at the same time, it's going to install all the dependencies for our front-end and back-end projects. It will start web servers and run automated tests all in this window. Now this is going to take a little while, so I'll be right back.

[07:47] Alright, our application is up and running and you can access it at localhost for 3000. So, here's what we get. We have a mini application for managing a list of movies.

[07:59] Now you know what's the beauty here? The beauty is that our database is populated with these movies as part of bringing up our application. I didn't have to manually insert these movies in our database. So we have a migration script for populating our database and Docker automatically executed our migration script as part of bringing up this application This is a very common real world scenario

[08:21] So here we can add new movies, movie one, movie two, whatever, and we can also delete these movies. So we brought up this application using a single command. Now I briefly mentioned this one,

[08:33] docker-compose.yaml. Before we talk about this line, first you need to understand the yaml format. This is a format that a lot of people are not familiar with. So in the next lesson, we're going to talk about JSON and YAML formats.

[08:52] Let's talk about JSON and YAML formats. If you know these formats well, feel free to skip this lesson. So in the root of this project, we're going to add a new file called data.json.

[09:04] Now JSON, as you probably know, is a language. It's a human-readable language for representing data. So in this case on 5, we can have an object or an array. Let's say we want to represent a course. A course can have properties like name, price, and so on.

[09:18] So in this object, we can add one or more key value pairs. Our keys should always be surrounded in double quotes. So we can add a key called name and set its value to, you can use a string, the ultimate,

[09:31] Docker course. Then we add a comma to define the next key value pair. So we can say price, we can set this to a number. Now the value can also be a boolean, so we can define another key value pair and set the value to true or false.

[09:48] We can define another key value pair and set the value to an array. So we define an array using square brackets. Now in this array we can have any valid objects. So we can have strings, numbers, booleans or other objects.

[10:01] So I'm going to add a couple of things, let's say software and DevOps. And one last key value pair. Author, I'm going to make this an object. So once again, use curly braces to define an object.

[10:15] In this object, we add a couple key value pairs. First name is Mosh, and last name is what? Hanadani. And yes, I am Iranian.

[10:28] I get that question all the time. Alright, so here we have a JSON file. Now let's see how we can convert this to YAML. YAML is another language for presenting data, but it has less clutter than JSON. It's easier to read. So I'm going to copy all this code.

[10:44] Here in the project, we're going to add a new file called data.yaml. The expression can be yaml or yml. Now, on the top, we add three hyphens to indicate the beginning of a YAML file.

[10:58] Then we paste our code. Now, in YAML, we don't use curly braces to indicate hierarchy. This idea has come from Python. If you are programmed in Python, you know that in Python, we use indentation to represent hierarchy.

[11:11] So we don't have curly braces. So, let's get rid of these braces. And remove the indentation. Good. Now, the next thing you need to know about YAML is that we don't have to use quotes.

[11:23] So we can bring up the replace dialog and replace all these double quotes with nothing. That immediately takes a lot of clutter away. Also, we're not going to use commas to separate key value pairs.

[11:39] So on the top we have name, price, if published. Now how do we represent a list or an array? We use hyphen. So I'm going to remove this. We press enter, add a tab on a new line.

[11:51] We type hyphen to define the first item in the list. software then at the same imitation we add the next item DevOps. Now author is an object but as I told you we don't use curly braces we use

[12:05] indentation so because these two properties are invented they belong to the author property okay so this is our yaml file let's compare this with JSON as you can see yaml is easier to read and

[12:19] understand. Now, why don't you use YAML all the time? Well, because parsing YAML files is a little bit slower than parsing JSON files, because the parser doesn't know if this is a string or a

[12:31] number. So you have to read everything as a string and then try to evaluate it. In contrast, in JSON, strings are represented using quotes and more specifically double quotes. So the parser knows that this is a string and it shouldn't evaluate it. Okay, so quite often you use YAML files for

[12:48] configuration files and JSON for exchanging data between multiple computers like a client and server. So now that you understand this format, next we're going to talk about compose files.

[13:04] Let's see how we can create a compose file from scratch. So for this lesson I'm going to rename this file to underline docker-compose. We want to set it aside

[13:16] and create a new compose file from scratch. So here we have a new file called docker-compose. All in lowercase, make sure to split it properly, otherwise docker-compose is not going to find this file,

[13:29] because this is the default name that docker-compose assumes. Okay? So, demo. Now, the first thing that we need to set here is the version property. What version should we use?

[13:41] Let's search for Docker Compose file. On this page, you can see various Compose file formats and their compatibility with Docker Engine. We are using the latest version of Docker Engine, so I want to use the

[13:56] latest Compose file format so we have access to the latest features. So we're going to set this to 3.8. Now, here we need to wrap this number with double quotes. otherwise it will be evaluated as a number but Docker Compose expects this

[14:11] value to be a string. Why? I have no clue. So here's the version. Now in this file we define various building blocks or services of our application. So we have a property called services. Now what services do we need here? Well our

[14:26] application has a frontend, a backend and a database. Now your application might have other moving parts. So you can define them here. Now, these names are arbitrary, so we can call them anything. We can change this to DB, we can change the backend to API,

[14:42] and the frontend to rep. The idea here is that we're defining various services and telling Docker how to build images for each service and how to run these images. So here we're going to have

[14:56] properties, and the value of these properties will eventually be used when running our containers. So in the previous section, we had to manually run our containers using docker run. And here we use parameters like dash p for port mapping or dash v for volume mapping.

[15:13] We also had to specify an image like React app. All these values can be defined in our compose file. So we don't have to manually start our containers. Docker compose will take care of starting our containers under the hood.

[15:27] So for each service, we need to tell Docker how to build an image for that service. So here we can use the build property and tell Docker Compose where it can find a Docker file. So if you look at this project, you can see that in our backhand and fronthand folders, we have a Docker file.

[15:46] This Docker file is almost identical to the one we created in the previous section. So we start from a node image, we create a user, we set up our working directory, copy all the files and install the dependencies, then expose port 3001 and start the web server.

[16:03] You have a similar Docker file in our frontend project. Let's have a quick look. So, that was the backend, here's the frontend, and here we have the Docker file. Almost identical. But the frontend application or the frontend server starts at a different point.

[16:19] That is the only difference. So, each service should have its own Docker file. Okay? Now, Back to our Compose 5 For our rev or frontend you going to set the build property to period meaning current folder slash frontend This is where you have a Docker file For our API

[16:39] we're going to set build to backend. Now for our database, we're not going to build that image. We're going to pull an image from Docker Hub. So instead of the build property, we're going to use the image property.

[16:51] Now for this application, I'm going to use mongo version 4.0-xenial. So that is mongo version 4 built on top of Xenial, which is ubuntu version 16.

[17:03] If you look at Docker Hub, you can see that mongo also has images built on top of Windows, but Windows images are very large, over 2 gigabytes. So that's why I prefer to use Linux images.

[17:15] So for any of these services, we can either build an image or pull it down. Now here we also have port mappings, so we set ports too. Now because we can have multiple port mappings, here we need to use the array or list syntax.

[17:28] So we use a hyphen and then define a port mapping. So a front-end application starts on port 3000. So I want to map port 3000 of the host to port 3000 of the container running this image.

[17:42] that's similarly for our API we're going to define a port mapping this one is going to be 3001 to 3001 now mongodb by default listens on port 27017 so i want to map the same port so we can access

[18:03] mongodb using a mongodb client like mongodb compass if you don't use mongodb you have the the same concept with other database engines. All these database engines listen on the default port. You want to map that port so you can connect your database engine using your favorite database

[18:20] client. Okay, what else do we have here? Back to this page for Compose file, if you look at version 3, on the right you can see all valid properties. Now a lot of these are for really special cases,

[18:33] so you don't need to use them all the time, but the ones I've used most of the time are build or We also use ports, volumes, environment, and so on. So our API project needs an environment variable that tells where our database is.

[18:49] So here we set environment. And here we can use the list syntax because we can have multiple environment variables. So we set db underline url 2. Here we need to type a mongodb connection string.

[19:03] These connection strings always start with mongodb, colon, two forward slashes. Here we need to type the name of a host. So as I will show you later in this section, when we start an application with Docker Compose,

[19:16] under the hood, a network is created. On this network, we're going to have three hosts. The name of these hosts are equal to the names we have defined here. So we're going to have a host called db. So that is the connection string for our mongodb server.

[19:31] Now, on this server we can have multiple databases. So, we're going to specify the database name and the conditional string as well. So, this is one way to set an environment variable. But, instead of using the list syntax, we can also use the object or property value syntax.

[19:47] So, we get rid of the hyphen. We say tburl is a property and this is the value of that property. property. I find the syntax more readable because we get color coding and it's just

[20:00] cleaner. Similarly we can add additional environment variables. Now we're almost there. The last thing we want to add here is a volume because we don't want MongoDB to write data to the temporary file system of the container. So here we

[20:15] set volumes and again we can have one or more volume mappings. So we add a hyphen. We're going to map a volume called Whitley. Of course you can call it anything. Whitley is the name of this

[20:30] application in case you didn't notice. So we're going to map this volume to a directory inside the container. Now if you look at the documentation of MongoDB on Docker Hub or just a typical

[20:43] MongoDB documentation, you know that by default MongoDB storage is data in slash data slash DB. So we want to map this volume to this directory. So whatever that is written inside this directory is actually outside of this container.

[20:59] It's somewhere else in our volume. Now because you have used this volume here, we have to define it in our compose file. So click on enter, remove all the indentations, so now we are at the same level as services.

[21:14] Here we are going to define another property called volumes. And here we're going to add another property called zidly with no value. I know this looks a little bit weird, but this is the syntax we have to follow.

[21:28] We just have to define the volume first before we can use it. So this is our compost file. Now, we can make this more readable by adding line breaks in between these properties.

[21:40] We can also order these services any way we want. So currently, I'm ordering them from front to back. We can also order them from back to front. So we'll put database first, then API, and then RAP.

[21:52] So we're done with our Compose file. Next, I'm going to show you how to build the images. Earlier, I told you that Docker Compose is built on top of Docker Engine.

[22:08] So everything we've done with Docker Engine, like building images, listing them, starting containers, and so on, all of these operations are also available using Docker Compose. Let me show you. So we type Docker Compose without any arguments, enter.

[22:23] Look, we have all the sub-commands like we have RM for removing stop containers, we have run, we have push, pull, and so on. The difference is that any of these commands will apply to our application as a whole.

[22:36] So most of these commands will impact multiple services or multiple containers in our application. So, let's look at Docker Compose Build and also use the Help option.

[22:49] So, we have a bunch of options here. A couple of them I want to point out that are useful to know is No Cache. We just can prevent caching when building the image. Sometimes, you encounter weird issues and you want to make sure that cache is not used.

[23:03] In that case, use this option. Another useful option is Dash Dash Pull. With this, we can always pull a newer version of the image. That is also good to know. So in this instance, I'm not going to use any of these.

[23:17] We're just going to run Docker Compose Build. This builds our web and API services. And as you notice, our build was super fast because pretty much everything came from the cache.

[23:29] So let's run Docker Images. So I have five images on this machine. We built the frontend, with the web, with the API, with the backend and Mongo. Mongo obviously came from Docker Hub.

[23:42] Now, as part of this build process in this lesson, we built Weedley web and with the API. These two other images, with the frontend and backend, were built when we started this application earlier.

[23:54] So, back to our project, in this original compose file that I included in this project, Look, I call these services frontend and backend instead of web and API.

[24:06] That is why we have these two images, bitly frontend and bitly backend. Also, as you have noticed, when building images with Docker Compose, our images are prefixed with the name of our application.

[24:19] Now, where does this come from? It is the name of the directory. So currently, we are inside a directory called Whitly, and that is why all those images are prefixed with Whitly. I think this is a great convention. Now I got a question for you.

[24:32] If you look at the created column, you can see all these images were created an hour ago. But didn't we just build the web and API images? Why do you think this happened? Here the answer Because I built these images frontend and backend an hour ago when I was recording the first lesson in this section Now when building these new images Docker used everything in the cache because all those files were already available all those layers were there so Docker didn have to do a full rebuild That is why we are still using the build from an hour ago

[25:03] okay now if you want to force a full rebuild you can say docker compose build dash dash no cache

[25:15] all right it's going to take a few seconds so i'll be right back all right our images are built so let's run docker images there you go look at the first two images hi and web were built less than a minute ago so that's all about building images

[25:31] Next we're going to talk about starting an application. You briefly saw how we can start an application with Docker Compose.

[25:44] We'll just type Docker Compose up. Now, if the images are ready, Docker Compose will run them inside containers. Otherwise, it's going to build the images automatically. Now, before executing this, let's look at the available options.

[25:58] So here we have a ton of options. A couple of them that are useful are build. With this we can force them to rebuild every time we want to start our application. So we don't have to explicitly run Docker Compose build and then up. We can combine the two using the build option.

[26:17] The other useful option is dash D for detached mode. So we will start these containers in the background. So, take a look. All right, now, if you run docker-compose-ps,

[26:31] we can see all the containers relevant to this application. In contrast, if you type docker-ps, we can see all the running containers across all applications. Okay, so here we have three containers,

[26:45] with Vibling, API 1, Bitly DB 1, and Web 1. And what is this 1? Well, we can start multiple containers from the same image. And this is used for high availability and scalability.

[26:57] It's something we'll look at in the future. So here you can see the container. You can see what command started that container. So for our API, that was npm start. For our database, that was mongod, or mongodamonprocess.

[27:12] And for our web front end, that was npm start as well. You can see all these containers are up and running. And over here, you can see port mappings. So now if you go to local host, port 3000,

[27:24] you can see our application. Beautiful. Now, how do we take this down? Let's say we're done with this application, and we're going to free up resources. Back to the terminal, we type dr.compose, down.

[27:37] This will stop and remove this container. But the images are still there. So next time you want to sort the application, the application will start pretty quickly. Let's talk about networking in Docker.

[27:53] Let's move on our application with Docker Compose. Docker Compose will automatically create a network and add our containers on that network. So these containers can talk to each other. Let's see this in action. So I'm going to bring up the application one more time in the detach mode.

[28:09] Good. Now look at the first line. Creating network with the default. So, you can run Docker Network MS. Here we can see all the networks on this machine. I think every Docker installation has three networks. Bridge, host, and none. Honestly, I'm not sure what these networks are for. But what matters here is that we have a network called Ridley Default. The driver for this network is Bridge on Linux or NAT on Windows.

[28:38] Now, this network contains three hosts or three containers. Web, API, and DB. So, these hosts or these containers can talk to each other using their name.

[28:51] Let's see this in action. So, back to the front row, let's look at the running containers. So, we have Mongo, Web, and API. Now, we're going to start a shell session on the web container and ping the API container.

[29:05] Take a look. So, we're going to execute in the interactive mode. The container ID is 8CS6, and we're going to run shell. So, let's paint API.

[29:18] We've got a permission error. Because we have logged in with the app user, that comes from our Docker file, remember? So, we have logged in with app user, and this user doesn't have ping permission. So, let's exit.

[29:30] I'm going to bring up the last command. Now, here we have to use an extra option for setting the user. We're going to login as the root user. Good. Now look at the shell prompt.

[29:42] You have a pound sign, which means you have the highest privileges. So here we can ping API. Now look, we're getting responses from the machine to this IP address. Now on your machine, this IP might be different.

[29:56] Now, let's start from Show SC to get out of this. So this is what happens under the hood. Docker comes with an embedded DNS server that contains the name and IP of these containers. Now, inside each container, we have a component called the DNS resolver.

[30:12] This DNS resolver talks to the DNS server to find the IP address of the target container. So, when we claim the API container, this DNS resolver asks the server, what is the IP address of the API machine, or API container.

[30:26] The DNS server returns the IP address and then the web container can directly track the API container using its IP address. Each container has an IP address and is part of a network. Let me show you one more thing. Back to the

[30:41] terminal, here we can run isconfig to see the IP address of this container. Take a look. This container has two network adapters. One of them is Ethernet 0 and over here

[30:54] you can see the IP address of this container. So 172.21.02 is the IP address of the red container. Now back to our Compose file, earlier when we defined the API servers, we added an environment

[31:10] variable that contains a database connection string. In this connection string, we have db, which is the name of a host. That is a db host or the db container. You saw that our API container

[31:22] can talk to this container because both these containers or all containers in this application are part of the same network. Now, one thing I want you to understand here is that this host is only available inside the Docker environment.

[31:36] So, if I open up my browser and go to localhost slash DB, I'm not going to get anything. So, the API container can directly talk to the DB container, but if you want to access this container,

[31:48] you need port mappings. and that is why we have this port mapping over here so this port on the host is not this port on the container so if you open up MongoDB contest which

[32:01] is a popular MongoDB client you can establish a connection to localhost port 27 017 because this port is not our container let's verify this real quick

[32:13] so it connects great so here we can see all our databases here's our future database and in this database we have a collection called movies with four documents so here are the movies that we currently have in the database so this

[32:27] is all about Docker networking next we're going to talk about viewing logs thank you so much for watching this video as I said this video is part of my ultimate Docker course that teaches you everything you need to know about

[32:41] Docker from the basics to more advanced concepts so if you want to learn more I I highly encourage you to take a full course. It's much faster and better than jumping from one tutorial to another. If you're interested, the link is below this video.

[32:54] Thank you and have a great day.