Queues in Laravel: Main Things You Need to Know (Two Examples)

[00:00] Hello guys, today let's talk about

[00:01] Laravel Q's. I will try to take this

[00:03] very complex topic and explain the main

[00:05] things that you need to know about Q's

[00:07] in roughly 10 to 15 minutes. And this is

[00:09] kind of a challenge I decided for

[00:10] myself. Take some complex topic in

[00:13] Laravel and try to explain it in 10 to

[00:15] 15 minutes. So you will see more of

[00:16] these videos in the future. How did I go

[00:18] with this simplification? Shoot in the

[00:20] comments below. So today we'll talk

[00:22] about how to put the email notification

[00:24] in a queue with this example and I'll

[00:26] show you how different the experience is

[00:28] with and without Q's. Then the second

[00:30] example would be how to create a Q job

[00:33] to build the PDF invoice for the order

[00:35] and the link to code repository for both

[00:37] examples will be in the description

[00:38] below. Also we'll talk about how to deal

[00:40] with Q job failures, how to generally

[00:43] run the Q and what should be the Q

[00:45] driver locally on your server and on

[00:47] production in a remote environment.

[00:49] Let's begin. First, when you would need

[00:52] cues in the cases where you don't want

[00:54] your users to wait, for example, for

[00:56] sending email notifications, generating

[00:58] some reports, or processing the uploaded

[01:01] files. In that case, you show the

[01:03] success message here on screen and then

[01:05] put the jobs in the background in the

[01:07] queue. And here's a simple example. You

[01:09] create a new task and you need to notify

[01:11] the users that are assigned to it. And

[01:14] then those users should get email like

[01:16] this. But it takes a few seconds to send

[01:18] that email. So let me show you the time

[01:20] difference and experience difference if

[01:22] you do it without Q and with Q in the

[01:24] code in the controller. We have this. We

[01:26] create the task in the store method and

[01:27] then we send the notification to the

[01:29] assigned users. But in this case I

[01:32] didn't enable the queue. So what happens

[01:33] if we open the network tab here and

[01:36] notify three users. Let's see how much

[01:39] it takes. Pending. Pending. A few more

[01:42] seconds and in total 4 seconds for user

[01:46] to wait. And imagine what happens if

[01:48] they need to notify 10 users or 20

[01:50] users. So this is the perfect case for

[01:52] Laravel cues. And in case of

[01:54] notification class in Laravel, it can be

[01:57] queuable very easily. In fact, it is

[01:59] queuable by default. I just disabled

[02:02] that. So in the notification task in the

[02:04] class, it needs to implement shoot Q and

[02:07] use ceable. That's it. So let's try

[02:10] again now. And by the way, those emails

[02:12] were successfully delivered. I'm using

[02:13] Mail Trap to test emails. So three users

[02:16] were notified. Now let's try with a Q

[02:18] create new task. I will use fake filler

[02:21] chrome extension. Let's notify all four

[02:23] users in this case. Create task and the

[02:26] result is immediate. So success on the

[02:28] screen for the user and then the jobs

[02:30] for notifications go where it depends on

[02:33] your setting of the Q driver in the

[02:36] default. ENV4 Laravel Q connection is

[02:39] database which means it uses database

[02:42] driver and the default Laravel comes

[02:44] with a few tables migrations for jobs

[02:47] batches and failed jobs. So now if we

[02:50] look at our database we have table jobs

[02:52] with four items each of them

[02:54] representing one notification email to

[02:56] one user. They are not executed yet.

[02:58] They are in the queue waiting for their

[03:00] time and then the developer needs to

[03:02] start processing the queue and you can

[03:05] do that with command PHP artisan Q work.

[03:08] If we launch that running then a few

[03:11] seconds to send the emails subsequent

[03:14] emails for some reason go faster and now

[03:16] in the mail trap we have four new emails

[03:19] and not only that if we leave Q work in

[03:22] the background it becomes a background

[03:24] process. Then look what happens if we

[03:26] create another task. We create the task

[03:28] and now if we go to terminal it's

[03:31] immediately catching the new jobs. So

[03:34] this is how the cues are supposed to

[03:36] work on live server and we will talk

[03:38] about that in a few minutes. There is a

[03:40] process in the background in the queue

[03:42] separately from the browser. So the

[03:44] browser still shows the success result

[03:46] quickly and then separately you need to

[03:48] maintain the Q worker to always work and

[03:51] always be ready for new jobs. And not

[03:54] only notifications can be queued just

[03:56] like that. So here's the dogs page for

[03:57] queueing notification but also in the

[04:00] same way you can cue mailables also by

[04:03] adding implements shoot Q and event

[04:06] listeners in the listener you can add

[04:08] implements should Q for the same

[04:10] behavior but what if you have some

[04:12] general process like for example sending

[04:14] a PDF invoice for that you would create

[04:16] a custom job so here's another example

[04:19] you create the order and you want to get

[04:21] the invoice PDF and that PDF may take

[04:24] quite a long time to build a few seconds

[04:26] again. So we create a separate job with

[04:28] artisan command php artisan make job for

[04:30] example generate order invoice and then

[04:33] in the handle method you do whatever you

[04:35] want and then put that job into the

[04:37] queue again the job implements shoot q

[04:39] andsqable and then in the handle the job

[04:42] is building the pdf from blade view for

[04:45] the invoice and to put that job into the

[04:47] queue we need to dispatch it from

[04:49] somewhere for example from controller so

[04:51] in the store method in addition to

[04:53] request products and order creation we

[04:56] have this line. So job class dispatch

[04:59] and you may pass the parameter if you

[05:01] need. So for example in this case we

[05:03] have a parameter of order ID which is

[05:05] automatically processed with PHP8

[05:08] constructor property promotion. So you

[05:10] don't need to create a property manually

[05:12] and then in the handle method you use it

[05:14] this order id and kind of a pro tip it's

[05:18] often beneficial to re-query the data

[05:20] from the database in the Q job because

[05:23] by the time the job is actually executed

[05:26] processed the data may have been changed

[05:28] in the database and as a result if we

[05:31] fill in the form create order look at

[05:33] our Q work which is still running it

[05:36] catches the job executes that and we

[05:38] have this PDF as a result also you may

[05:40] want to send that PDF right away. So for

[05:42] example in the same job you may call the

[05:45] notification. Now what happens if the

[05:47] job fails for whatever reason some SQL

[05:49] query error or typo in the code or

[05:52] something like that. So the user is

[05:54] notified about the success but then the

[05:55] background job isn't executed

[05:57] successfully. Let's simulate that

[05:59] scenario and I will show you how it

[06:00] works for database driver first locally

[06:04] and then it's a bit different for

[06:05] production servers with different

[06:07] driver. We'll talk about drivers in a

[06:09] few minutes. So for example, in the job

[06:10] let's make a typo intentionally model

[06:13] that doesn't exist. Now we execute the

[06:15] same thing. Create order. Still success.

[06:17] The order is actually created, but the

[06:20] background job in the terminal is marked

[06:22] as done. And this is another important

[06:25] thing you need to understand about the

[06:27] cues. If you change something in the

[06:29] code, Q work doesn't automatically

[06:32] restart and take the changes of your

[06:35] code. It works with the Laravel version

[06:37] at the time when that Q work was

[06:39] started. So in our case, it actually

[06:41] succeeded. To simulate the failed job

[06:44] we need to restart the queue. So let's

[06:46] stop that one and launch Q work again.

[06:49] It will listen again for the jobs, but

[06:52] now with this version of Laravel code.

[06:54] So let's create another order. And now

[06:56] what we will see is fail. This is

[06:59] exactly what I wanted to show you. So

[07:01] there was a job in the table of jobs.

[07:04] But now if we refresh it's empty which

[07:06] means the job is executed. But if it

[07:09] fails two things happen. First you can

[07:12] find the actual error in the log

[07:14] laravel.log. So this is the actual error

[07:16] like you would find any other error of

[07:18] Laravel. But also how do you now retry

[07:21] the job because it's not in the jobs

[07:23] table anymore. So this is where another

[07:26] database table comes in. Failed jobs. It

[07:29] is logged here now with the actual error

[07:32] exception. And if you want to retry the

[07:34] job, for example, you fix the error. So

[07:36] let's get it back. Then we need to

[07:38] restart Q worker again. Q work again.

[07:40] And this is what you need to do every

[07:42] time you deploy new code to production.

[07:44] You need to restart the Q worker as a

[07:46] part of your deployment process. And now

[07:49] our goal is to get that job from failed

[07:51] jobs to the jobs table. And for that in

[07:54] the separate tab, we will execute a

[07:55] command Q retry. There are a few

[07:58] parameters of job that you can retry.

[08:00] For example, you can provide the ID of a

[08:03] job. But in this case, I will just

[08:05] choose all and it will retry the jobs.

[08:08] So they are back from failed jobs to

[08:11] jobs. But actually it is not here

[08:13] because it's already automatically

[08:15] executed, done and not failing anymore.

[08:18] So this is how you generally manually

[08:20] retry the jobs. But also the jobs

[08:22] themselves have a few parameters to

[08:24] specify how many times they should be

[08:27] retrieded. What are the parameters for

[08:29] max exceptions, timeout, it's so-called

[08:32] back off, a lot of parameters that you

[08:34] can read in the docs of Laravel. Now

[08:36] let's talk about cues on production

[08:38] server in live environment. This is

[08:39] different than local database driver.

[08:42] Here are other drivers available in

[08:44] Laravel Q's. And the most popular one

[08:46] and the most typical one used in Laravel

[08:49] is Reddis. So to set up the cues on

[08:51] production, you probably need the

[08:53] combination of these tools. Reddis

[08:55] here's the homepage. This tool is not

[08:57] from Laravel or from PHP. In general

[08:59] it's much more complicated. And we have

[09:01] a tutorial Reddus and Laravel 101 with

[09:04] general overview not for Q specifically

[09:07] and I will link that in the description

[09:08] below. But as a part of that tutorial

[09:10] we have set up for Q connection Reddus.

[09:13] And then that would be the overview of

[09:15] Reddus itself. But this is where another

[09:17] tool comes in which is Laravel Horizon.

[09:20] This is a dashboard for Q jobs usually

[09:22] for production server which works

[09:24] specifically with radius. As an example

[09:26] we had a tutorial about Q performance

[09:29] test sending a lot of PDFs and when we

[09:31] set up the queue with Reddis. Later in

[09:34] the article, we could monitor how many

[09:36] jobs were processed and the runtime and

[09:39] the dashboard of jobs per minute and so

[09:42] on. Solar Horizon is a must-have and a

[09:45] very powerful tool to monitor the queue

[09:47] jobs, but also a few commands like

[09:49] restarting the queue should happen not

[09:52] with manually Q termination from

[09:54] terminal or command Q restart but a

[09:57] specific command PHP artisan horizon

[10:00] terminate which would take care of all

[10:02] of the Q workers in the background and

[10:04] that background is also provided by a

[10:06] third tool that you need to know called

[10:08] supervisor which is again not from

[10:10] Laravel and not from PHP. P it's a

[10:13] monitor or process control system to

[10:15] restart the Q workers automatically if

[10:18] some of them fail cuz not only job can

[10:21] fail but also the Q worker process may

[10:23] crash for whatever reason and in the

[10:25] Laravel documentation there's a specific

[10:27] section about supervisor configuration

[10:29] how to install it and the options that

[10:32] you may choose and if that sounds

[10:34] complicated and it is you may install

[10:37] all of that much easier with help of

[10:39] tools like for example we're using

[10:41] Laravel Forge in many of our projects.

[10:43] It allows to just enable radius

[10:45] connection for Q with just a few clicks.

[10:48] Also providing the supervisor

[10:50] automatically so you don't need to care

[10:52] about how it works internally. Also, you

[10:54] can do that with Laravel cloud or not

[10:56] first-party tools like ploy for

[10:59] deployment. Disclaimer, I'm not

[11:00] affiliated with any forge or cloud or

[11:02] ploy or whatever. I genuinely think that

[11:05] setting cues up manually on VPS is just

[11:08] not worth the time. So these are the

[11:09] basic things about how Q works that I

[11:11] wanted to explain in this video, but

[11:13] there's so much more to it. The official

[11:16] Laravel documentation is pretty long

[11:18] with a lot of cases. And this is where

[11:20] the main goal is when working with cues.

[11:23] Manage them, monitor their performance

[11:26] restart the jobs, restart the Q workers

[11:28] scale the Q workers into multiple Q

[11:30] workers on maybe separate server. So

[11:33] there's a whole new world how to

[11:34] optimize cues for really big scale. On

[11:37] Laravel daily, we have two courses about

[11:39] Laravel Q's just recently updated cues

[11:41] in Laravel 12 kind of for beginners. So

[11:44] it's this video and on top sections like

[11:46] longunning jobs with timeouts. So, we're

[11:49] getting deeper into the same topics that

[11:50] I mentioned in this video. You can see

[11:52] the chapters on the right, but also

[11:54] there's a second course called practical

[11:56] laral cues on live server which is a bit

[12:00] older almost 2 years ago, but the

[12:02] fundamentals didn't really change. Same

[12:04] supervisor, Horizon, and Reddius. So, I

[12:07] will link both of those courses in the

[12:08] description below. What do you think

[12:10] about Q's? Have I missed something

[12:11] important or what questions do you have

[12:13] that I could cover in separate videos?

[12:15] That's it for this time and see you guys

[12:17] another