I Compared Every Major CI/CD Strategy So You Don't Have To

[00:01] e-commerce platform and your team just finished building a new checkout feature. With this new feature, customers can now save multiple payment methods and there is a buy now button that skips the cart entirely. You've

[00:13] tested it on your laptop, everything works. So now you need to deploy it to actually use it. But here's the question, do you deploy directly from your development environment straight to production or do you first deploy to a

[00:27] test environment, then staging environment, then maybe pre-production, Different teams have completely different answers to this question and a lot of them think that other approaches

[00:39] are wrong and what they're doing is the correct way. Some teams deploy straight to production with just a code review. Other teams have five environments between development and production and it takes two weeks to deploy a simple

[00:51] change. So in this video, I want to show you the three main CI/CD pipeline strategies that I see in real companies and real projects. I'm going to explain and real projects. I'm going to explain why each one exists, when each one makes

[01:04] sense, and help you choose the right approach for your team. Because here's the important point, there's no universally correct number of environments. If there was, then we would just have one CI/CD strategy and

[01:16] that's it. But it really depends on what you're building, how fast you need to deliver, and most importantly, what the cost of a bug in production is for your project or company. So let's start with the fastest, most aggressive approach.

[01:33] So strategy one is the minimalist, development to production straight away. So let's go back to our e-commerce platform where your team decides, we're development straight to the end users, no intermediate stages. So here's how it

[01:47] works. You write the checkout feature code on your laptop, you write automated tests, unit tests that verify that the logic for saving payment method actually works. Then you write integration tests that verify that the buy now button

[02:00] correctly creates an order. And once you're done, you push your code to Git repository. The CI/CD pipeline gets automatically triggered. It runs code quality checks, security scans, and if everything passes, your code is

[02:13] automatically deployed to production. So, from your laptop to live customers in minutes. Now, some of you may be thinking, "Yeah, that's the way to do it." But a lot of you may be thinking, "Why would anyone do this? This seems

[02:25] risky, completely irresponsible, deploying straight to production." But here's a thinking behind it. If your automated tests are really good and reliable, you don't need manual testing environments. So, let me explain with

[02:37] our checkout feature. You've written tests that verify payment methods are now button creates orders with the right data, edge cases are handled, what if someone clicks buy now twice, or does some other weird stuff that users

[02:50] integration with the payment processor works. If these tests all pass, the code should work in production. So, why wait and introduce manual bottlenecks in between? So, the core philosophy here is that if you invest heavily in proper

[03:06] automated testing, then you're going to trust your tests to test everything production. So, instead of spending time maintaining staging environments and doing manual testing, you spend the time writing better, more complex automated

[03:20] think of it like when cooking, some people taste their food multiple times while cooking. After each ingredient or every time they put spices in it, they taste and adjust after each set. The minimalist approach is like following a

[03:34] precise recipe with exact measurements. And if you follow the recipe correctly, then you don't need to taste every step. You know that the dish is going to be fine at the end. But as I said, every strategy fits specific type of projects.

[03:48] So, this approach works especially well for startups that are moving fast quickly. You probably don't have tens of millions of users, so your main priority is to basically iterate really fast based on user feedback. So, waiting 2

[04:03] weeks for a deployment may kill your momentum and your entire startup. It also works really well if you're building tools for your own company, customers are your co-workers. So, if something breaks, they can tell you

[04:15] immediately and you can fix it. There's not going to be a really dramatic event work properly. And obviously works for teams of any projects that have excellent test coverage. So, if you have 80% plus code coverage, but also very

[04:29] important, the quality of tests are reliable, so they test very complex scenarios end to end, you can trust the tests. And finally, you can deploy code to production with feature flags, where the features that are in development or

[04:43] in progress are kept hidden for the end users, but it's turned on for just 1% of users first to see if it actually works, nothing breaks, and then gradually increase if everything looks good. However, there are a few issues that you

[04:56] may run into when using this approach. First of all, you may realize your tests are actually not as good as you thought they were. So, you wrote tests for many scenarios, you're happy with it, but you forgot to test some edge cases, or maybe

[05:09] you forgot to test what happens when payment processor is down. So, all your happy path tests actually pass, you deploy, and suddenly production breaks when Stripe has an outage. And then you jump into a panic mode fixing the bug in

[05:22] the production. And when that happens, your customers basically become your QA team, because when bugs slip through to the production, the actual customers hit them. They get frustrated, they complain on Twitter, they complain on Reddit or

[05:34] X, and your support team gets flooded with tickets. And there is one not obvious thing that sometimes happens in teams that use this approach, which is the pressure not to break things slows you down. So, because everyone knows

[05:49] that each commit will land directly into the production, it slows down or makes the team actually extra careful about every single commit. So, ironically, deploying straight to production can make teams way more conservative. So,

[06:01] people become afraid to commit when the deployment is risky, which kind of defeats the purpose of the strategy. So, the minimalist approach is fast, but it requires discipline and really excellent automated testing that is really good in

[06:15] reality, not just in your head. And many teams think they have good tests and they're ready to go until they try this approach and realize they don't. But, let's look at the exact opposite approach, which is the paranoid with

[06:28] five-plus environments. So, imagine a completely different team working on the same e-commerce checkout feature, and the team says, "We cannot afford bugs in production. It's going to kill our credibility. We have too many

[06:42] users on our platform. We cannot afford that." So, every bug costs real money: lost sales, lost customer trust, and sometimes potential legal or regulatory environments between development and production. They have dev and test and

[06:56] staging and pre-production and prod, and maybe they squeeze another one in just to be sure and just to be on the safe side. So, if we follow our checkout happens is you finish coding on your laptop and deploy to the dev

[07:10] environment. This is the first shared environment where your code that you wrote locally now runs on an actual server, not your laptop. The dev environment catches basic integration issues, right? Maybe your code works on

[07:22] your Mac, but fails on Linux servers. Maybe you forgot to add an environment variable. So, dev environment will catch this. Next, your code goes to the test environment. So, once it passes the dev stage, it basically moves or advances to

[07:35] the test environment. And this is where QA engineers manually test your feature. thoroughly, right? They try to break it. They try to do all types of weird things that they think users may also do. Try all the edge cases. What happens if they

[07:50] invalid credit cards? Does it work on mobile browsers? And so on. And they may spend hours just testing every single scenario. Usually, they find bugs and and redeploy to test. So, it goes through the same cycle. Once your

[08:06] feature passes the manual testing on the test environment, your code moves to staging. This environment is a clone of production. Same database size, same number of servers, same configuration. So, in staging is where you test

[08:20] performance and integration with production data. You run load tests. So, what happens if 10,000 customers use the checkout at the same time? Maybe it works for 100 customers, but it completely breaks under load. So, you

[08:33] test that. You also test migrations here. If your checkout feature requires database schema changes, for example, you test the migration process in staging first. Once everything is fine there, then your code goes to

[08:46] pre-production. And this environment is even closer to production. It might even use the same actual database as production servers. Maybe just in read-only mode. So, pre-production is basically a final sanity check. You test

[09:00] verify that the monitoring and alerts work properly. You make sure rollback procedures work. So, basically, any single imaginable scenario is tested here. And finally, after passing through all these environments, your code

[09:14] reaches production. It's been 2 weeks since you wrote code, or maybe even more, but you're very confident at this point that it works correctly because environments. Now, again, a lot of

[09:27] you should release anything, but a lot of people would ask, "Why would anyone do this? It seems like a waste of time and waste of engineer resources." It environments before production, but here's a logic behind each environment

[09:42] catches different types of problems. An analogy here would be an airport security. You might think going through multiple checkpoints is an overkill, but each checkpoint checks different things. One checks your ticket, then the next

[09:56] one checks your ID or passport, another one scans your bags, then there is another one that does body scanning. So, multiple layers that each check for different things for the same person. And as I said, there are completely

[10:08] valid use cases for each approach. For this one specifically, think of banking and financial services. The main difference here is that a bug in production may cost a lot of money for the institution itself, but also for the

[10:21] customers. So, a bug that loses customer money or exposes financial data can be catastrophic. So, the cost of slow deployments is actually worth the safety. The same way in healthcare systems, for example, a bug medical

[10:35] record system could actually harm people's lives. So, you need multiple enterprises with compliance requirements. You need audit trails, approval gates, verification at each step for regulatory compliance. So, it

[10:48] may not be even code logic and functionality that gets checked as much as is the code or the application compliant? Does it go through security checks and so on? So, generally, as you see the pattern is that systems where

[11:01] see the pattern is that systems where bugs cost millions or cause regulatory issues actually justifies having multiple testing environments, even though it may be painfully slow to deploy any new change. But, of course,

[11:13] it comes with lots of downsides, which is it takes forever, because imagine you're deploying a button color change and it's been sitting there in test environment for a week waiting for QA approval. And another problem may be

[11:25] that environments drift from production, because when you have to maintain five deployment environments and keep them in sync, that's a huge overhead. So, very often in practice, what happens is your staging environment maybe was a clone of

[11:38] production 6 months ago, but a lot of things changed on production. Different database sizes, different traffic patterns, you did some security patches, you added some servers, and staging does not represent production anymore. And

[11:52] it's also more expensive because you have to actually create and pay for those extra environments, servers, database, maintenance, monitoring. So, for a large application, this could be tens of thousands of dollars per month

[12:05] just for maintaining these extra environments. So, the paranoid approach is safe, but it's slow and expensive. So, we saw these two approaches for deployment, which are kind of two extremes of each other. However, in

[12:17] practice, most teams actually end up somewhere in the middle. practical. And this is what most pretty They use three environments: dev, staging, prod. So, if we follow our

[12:32] checkout feature through this pipeline, what happens is you finish coding and shared environment where the team can see and test new features before they're ready for customers. And again, the

[12:44] basic integration testing is done here to see if the feature actually works when connected to the database, to the actual APIs, and so on. And you also use dev for collaboration. So, your product manager may see the new feature and give

[12:57] you feedback about the usability, user-friendliness, and stuff like this, not just the functionality. And your designer can verify that it matches the actual designs. Once the feature looks good in dev, it moves to staging. And

[13:09] here's a key difference from the paranoid approach. Staging is specifically a production clone for testing the scary changes. So, what are scary changes? Database migrations that modify millions of rows. That's a scary

[13:22] change. Or major architectural changes, switching payment processor, for example. Or code refactoring, where multiple parts of the code actually changes to critical flows like checkout or login or registration. So, for our

[13:36] checkout feature, you test in staging whether the database migration that adds a new table for new payment method is actually working. Or you test integration with payment processor under realistic load. And you also test the

[13:51] deployment process itself. Any underlying dependencies that your code changes actually need. But here is what you don't do in staging. You don't manually test every single feature. Your automated tests already verify that the

[14:05] testing things that automated tests cannot verify, like deployment process, database migrations, load testing, production-grade load for the staging looks good, you deploy to

[14:19] production. And here is where the practical approach adds safety in this deployment phase. You deploy gradually. You can use techniques like canary deployments, where you deploy to 5% of your servers first, you monitor for

[14:32] errors, you just let it run for hours or maybe days, and see if everything looks good. Then you deploy to 20% of the servers, and then you increase that to 100, as long as nothing breaks. Another technique is feature flags. So, the

[14:45] checkout feature gets deployed with feature flags, which means by default switch, so it's kind of switched off by default, and you can switch it on or turn it on for maybe just 1% of customers first, and see what the

[14:58] feedback is. Everyone using the feature without issues, then you increase it to 10% of the customers, and you turn it up to 100% of the customers. And the final safety lever that teams often use with this practical approach is monitoring

[15:12] and rollback. So, you have monitoring set up that automatically checks within the first hours, first days, first week of the new deployment to see if there are any issues. And if error rates spike, for example, it automatically

[15:25] rolls back to the previous version. So, even though you are deploying with way less testing compared to the paranoid approach, you still have these guardrails that protect you from completely making a mess in case you

[15:37] deploy something that's broken. So, the practical approach kind of balances speed and safety. So, it's fast enough to deploy daily. You're not waiting weeks for changes to go through multiple environments. So, a typical feature

[15:49] might move from dev to staging to prod in two to three days, but it's safe enough to sleep well at night once you hit that deploy button. Because staging catches major issues and gradual rollout catches production-specific issues

[16:03] monitoring, it means you know immediately if something breaks. And the biggest upside here is that it's affordable. You just have two extra environments that don't cost as much as four, and staging does not need to be as

[16:17] large as production if you're just testing deployments and not performance at scale. And this approach actually works for majority of use cases. Most software companies like SaaS products, web applications, mobile backends, this

[16:30] can be a standard approach for these type of companies. But also teams that want to ship frequently but still have low-risk deployments, you can deploy with this multiple times per week while still feeling safe about each

[16:43] deployment. And for any customer-facing application where bugs are not inconvenient, but they're not like dramatically bad, like e-commerce, type of stuff, it's also a perfect approach for this. You still have some

[16:58] issues and risks with it, obviously, because the two additional stages still may drift from production. And of exactly match production, there are less things that you can test that is really

[17:14] realistic. So, you end up still doing a lot of the load testing or integration That's why we have the gradual rollout. And you still need good automated tests for this because staging does not replace testing. It's actually an extra

[17:28] step. So, if your automated tests are weak, then you're going to have a lot of bugs that reach production even with these extra two stages. So, now that you understand all three approaches to CI/CD deployment and why

[17:43] each one makes sense in which type of projects, I think you can answer this specific question yourself, which is how do you decide which approach is right for your team and your project. The simple formula is if a bug in production

[17:56] will cost the company a lot, then you need to be a little bit more paranoid. But if you're a fast-moving startup with relatively low number of users, then you can be risky especially at the beginning. And for any mature project

[18:10] that has a lot of users, but the cost of a bug in production is not dramatic, which is the majority of the projects, can use the practical approach. That's why it's called practical because it works for most use cases in practice. My

[18:23] personal recommendation, if you're unsure where to start, is always start with practical and then you can shift up or down depending on whether you decide, much faster." Then you go to the minimalist approach. But if you see,

[18:38] "Okay, there are too many bugs that are ending up in production that is costing our company reputation, that is frustrating our customers. We cannot afford that." Then you can introduce additional stages.

[18:52] Now, before we wrap up, I want to show you a few common mistakes that I've seen in practice and you may recognize yourself in one of them, which can be very practical for a lot of teams. Mistake number one is too many

[19:04] environments that nobody uses. I've seen companies with six environments and nobody can explain what makes pre-prod different from staging, for example, and why they need both. They added environments some years ago for reasons

[19:17] nobody remembers anymore, and now they maintain all six, and deployments take a month. And here's a formula to avoid this. Audit each environment, and for each one, answer this one question: Is this environment catching a bug or

[19:30] increasing our confidence of deployment? If not, remove it. Mistake number two is where staging does not match production or is not even close to production. And this is probably the most common one that I've seen, where staging

[19:44] as production, so there was a match at one point, but over time, it completely libraries, different configurations, different environment variables, different data volumes, so it kind of makes it useless because testing in

[19:59] doesn't tell you anything about how the so you're actually testing without purpose. Because the purpose of testing does not break in production, which means, naturally, you should test in a

[20:15] how do you know that it's going to behave the same way? So, the solution here is either keep staging synchronized with production or use staging just for basic integration testing, but also

[20:29] become aware and realize testing on staging does not give you any data about how the application will perform on a production environment. This one is very clear, where teams do not have automated tests, but they jump into minimalist

[20:43] approach. And it happens because of practical reasons, where a lot of teams think, "We have great tests. I think we're good to go." And besides, Netflix deploys straight to production, so should we. And here, obvious solution is

[20:56] just to do a reality check, whether you really trust your automated tests and whether your tests validate complex end-to-end processes in your application, not just simple happy scenarios. So, you started the practical

[21:09] approach, you add the automated tests until you feel fully confident, you add monitoring and some guardrails, and then move to the minimalist approach. And interesting one, which is adding environments instead of fixing tests.

[21:24] Very often, when teams have flaky, unreliable tests and bugs keep reaching production. So, basically, instead of writing good automated tests, they add environments and manual testing, which slow down the process. So, the better

[21:37] fix is always investing test quality, improve coverage. Good tests are always they cost less, you don't have to environments, and they don't take hours like manual testing, they're much

[21:50] faster. And that's a wrap up for this video. I hope you learned a lot here, and I hope that you could take at least one practical action step for your project and share it with the team. Or, if you have some other approach that I

[22:03] haven't covered here, that is super custom and interesting, please share it going to be fun to see what everybody's using in their real projects. And with that, thank you for watching, and I'll see you in the next video.