[0:00] Benchmarking PHP code is fun and it has
[0:02] always fascinated me, but it's also
[0:05] tricky and there are more ways to do it
[0:07] wrong than you can count on two hands.
[0:10] And if you're benchmarking wrong, then
[0:12] the results are at best useless, at
[0:15] worst harmful. And at the end of this
[0:18] video, you have learned a list of
[0:19] preconditions for good benchmarking in
[0:22] PHP. One tool that has helped me
[0:24] immensely in designing good benchmarks
[0:26] is Hyperfine. And I want to show you how
[0:28] it works, what its benefits are, and
[0:30] what problems you still need to account
[0:32] for. Mo, I am Benjamin and I have helped
[0:36] thousands of developers with PHP
[0:38] performance over the last 10 years. What
[0:40] is Hyperfine? You can write benchmarking
[0:43] code easily in PHP without additional
[0:46] help using HR time function or more
[0:50] commonly seen the micro time function.
[0:53] After doing this for myself for years,
[0:56] my colleague Fulker convinced me to give
[0:58] Hyperfine a try and it stuck. Why even
[1:02] use a dedicated tool for benchmarking?
[1:04] Fulker wrote about Hyperfine in our blog
[1:06] and we are going through his work and is
[1:09] examples to understand why Hyperfine
[1:11] helps with avoiding a bunch of
[1:13] benchmarking problems that are very
[1:15] common but also introduces some. You can
[1:18] install uh Hyperfine easily on a Mac
[1:21] with brew install Hyperfine and on many
[1:24] other systems as well with a package
[1:26] manager. And if you have it installed,
[1:28] you can prefix any command with
[1:30] Hyperfine. For example, in this case for
[1:32] the bench sleep script which runs u
[1:36] sleep um for 100 milliseconds
[1:43] and then you can also run it uh for
[1:46] multiple commands. For example, PHP
[1:48] bench sleep, PHP with a factor of two.
[1:52] And then again PHP bench sleep
[1:56] with a factor of three. And then it will
[1:58] run each of them individually and it
[2:01] will compare the performance against
[2:03] each other. This way you can run
[2:05] alternative implementations for your
[2:07] code and then benchmark them against
[2:10] each other and see which one is faster,
[2:11] which is slower and pick one of um the
[2:15] solutions that you want to use. Uh we
[2:18] use it for example also for testing the
[2:21] performance of different PHP versions
[2:23] against each other. Let's say if then
[2:25] improve is there an improvement in a
[2:27] version and then tests the same code
[2:29] against multiple versions of PHP and see
[2:32] how the performance changed. So what
[2:34] problems do exist with benchmarking? Now
[2:36] I want to talk about seven different
[2:38] things that you should account for when
[2:40] benchmarking PHP code.
[2:44] The first problem we should look into is
[2:46] not using a production equivalent
[2:49] environment. And this can happen because
[2:51] we are running the benchmarks uh using
[2:53] the PHP CLI command. This means that by
[2:57] default opcache is not loaded or not
[3:00] enabled for the CLI. And this can
[3:03] drastically change the performance um
[3:06] compared to running the same code in a
[3:08] web server. So let's check that Z
[3:11] opcache is enabled and also in get
[3:18] opcache enable CLSI is set otherwise
[3:23] throw new exception
[3:27] opach not loaded.
[3:31] So let's run
[3:36] the code again. we see oh okay opache
[3:39] was not loaded so let's change
[3:45] the ini for the cli settings we load
[3:48] opcache
[3:50] and next run works
[3:52] uh what we also should account for is
[3:54] that xdebug is not loaded so this is
[3:57] something where I had like several
[4:00] embarrassing moments where I tested for
[4:03] example I think PHP P 5.6 against PHP 7
[4:09] or so and I saw it so much faster PHP 7
[4:14] and then realized not only of course
[4:17] PHP7 was much faster than PHP56
[4:20] but I was off by a big factor because on
[4:23] PHP 5.6 I also had XD debug running and
[4:27] I didn't have that on um
[4:32] I didn't have that on PHP7. So uh if PHP
[4:36] XD debug uh either either it's not
[4:38] loaded or throw new exception XD debug
[4:43] is loaded and then we run that and this
[4:48] is something that uh specifically with
[4:50] benchmarking can happen because it's a
[4:52] system you run them on a system where
[4:54] you also develop. So it might be the
[4:57] case that XDbug is actually on. So here
[5:00] it's off um on my machine and that's
[5:02] fine.
[5:04] We could also account for other things
[5:06] that are non-production ready. It
[5:07] depends on your environment um what what
[5:10] you think of what you should account
[5:11] for.
[5:14] The second problem that you can have is
[5:16] writing PHP code that opcache optimizes
[5:19] away. So let's say we have um code here
[5:24] that has a function fu returning one and
[5:29] um we want to test if adding a function
[5:32] returning this is faster than just
[5:34] iterating the code here. So this is the
[5:37] original code that we want to test and
[5:40] um we are iterating this and dumping at
[5:43] the end and then we have also the code
[5:46] where we call the full function and we
[5:49] compare that against each other. So
[5:51] let's run PHP
[5:55] bench
[5:56] function loop PHP against PHP
[6:01] bench function
[6:05] optimized away.php and the the file name
[6:08] already gives away the problem. Uh
[6:11] what's happening here is that with
[6:13] opcache
[6:15] some of the functions are actually
[6:17] inlined and optimized away. So because
[6:20] the fu function is returning a static
[6:22] value
[6:24] um it will actually compile the code
[6:26] down to look exactly like this. Um and
[6:30] that means that you need to be careful
[6:32] with certain functions that you don't
[6:35] write them in a way that opcache
[6:37] optimizes them for your benchmarking
[6:40] scenario and in production you're using
[6:43] a different function. you write it a
[6:45] little bit differently or it's more
[6:47] complex because that's just the way
[6:49] production code is. So um that way you
[6:52] would see that um the performance is
[6:56] quite different. So what we found out
[6:58] here is that actually the code for um
[7:03] the function optimized away ran faster
[7:06] than the code um that's just
[7:09] incrementing the number. So if we relied
[7:11] on that benchmark, we would think that
[7:13] functions in PHP code are faster than
[7:16] just incrementing an integer, which is
[7:18] not true. Opcimize this away. In this
[7:22] case, we could fool OPC cache um into
[7:24] thinking this function cannot um uh be
[7:28] inlined
[7:30] by writing some dummy code that it gets
[7:34] confused by. And if we rerun the
[7:36] benchmarks now then we see the loop is
[7:39] at 100 milliseconds and then the
[7:41] function call is uh much slower. The
[7:44] factor is it's a factor of two and this
[7:47] is way more realistic. Now
[7:51] the third problem is not accounting for
[7:53] load on the machine and this happens
[7:55] with developer machines all the time.
[7:58] So, my developer machine has Docker
[8:00] running, has Spotify running, has a
[8:03] browser running, open with 10,000 tabs.
[8:05] Yes, I'm a tap messy. And it is usually
[8:09] under quite a bit of load. In this
[8:11] example, we see what's uh happening
[8:13] under load. the benchmark can be off
[8:17] because the um uh the machine actually
[8:21] doesn't have enough CPU to run a
[8:23] benchmark um on a CPU core without
[8:27] getting interrupted. And Hyperfind
[8:29] checks for interruptions. It checks for
[8:32] statistical outliers and it warns you
[8:35] about this. If there are outliers, if
[8:37] there are problems, it um explains to
[8:40] you that you should run it on a quieter
[8:42] system. And this is very helpful because
[8:44] you can now run the benchmarks as long
[8:48] as
[8:50] um
[8:53] uh you have a run that doesn't show a
[8:55] warning of this kind. Um maybe you need
[8:58] to stop a few programs for this to
[9:00] happen. Um but you will know that like
[9:03] the benchmarks here are not
[9:04] statistically valid because um there was
[9:08] a big like spike or change.
[9:12] The third problem you can run into when
[9:14] benchmarking, at least with Hyperfine,
[9:16] is that you don't account for the
[9:18] startup time of PHP itself. So let's run
[9:22] a PHP script through Hyperfine that does
[9:25] absolutely nothing. So it's PHP
[9:27] empty.php. We run it and we see that
[9:32] running this empty script takes 50
[9:34] milliseconds. And this means that
[9:37] running any PHP script like with this
[9:40] PHP um runs 50 milliseconds needs 50
[9:43] milliseconds of startup time and we need
[9:46] to discount that from all the benchmarks
[9:48] that we are running. So maybe you
[9:50] remember our benchmark from the
[9:51] beginning with the sleep. We can run it
[9:54] again. PHP bench sleep.php with a factor
[9:58] of one. We just had 100 milliseconds of
[10:00] US sleep.
[10:03] And as we can see the script is actually
[10:06] not running 100 milliseconds. On average
[10:08] it's running 167 milliseconds. And this
[10:11] is because of the startup time of PHP
[10:14] itself. And you need to discount that uh
[10:17] to be able to compare the different runs
[10:19] with each other. What we usually do is
[10:23] we um make sure that the benchmark
[10:26] script uh runs at least 500 milliseconds
[10:29] or better like around 800 milliseconds.
[10:32] so that the actual startup time is um
[10:35] becoming a very small part of this
[10:38] runtime.
[10:41] The next problem that you can have with
[10:42] benchmarking is not repeating the tests
[10:45] and this is a problem that uh Hyperfind
[10:48] accounts for automatically. Um as you
[10:51] remember uh when we run hyperfind for
[10:54] example on our
[10:58] um function bench function loop.php.
[11:03] So what we do here is uh 10 million
[11:08] iterations of an increment
[11:12] then it will run this 15 times and um
[11:16] Hyperfine looks at the execution length
[11:20] of the first test and then determines
[11:23] how often it should run them to be able
[11:25] to make a statistically good um estimate
[11:29] of the performance. And the reason is
[11:31] you should account for this is first um
[11:36] to detect outliers on the machine. So
[11:39] with this we are able to detect that one
[11:41] run took significantly uh longer like
[11:44] this one here. And this might
[11:46] potentially mean that the machine is
[11:47] under load and the benchmark is not
[11:49] reliable. But also you cannot just take
[11:53] one um
[11:56] um test and then use this number as the
[12:00] truth. That is not statistically um good
[12:03] and you can be way off by not accounting
[12:06] for changes that um or variances in
[12:09] this. Maybe you could argue that um
[12:12] running 10 million iterations is already
[12:14] repetition, but uh for me it's really
[12:17] not. The problem is that you want to see
[12:20] that the performance runs the same when
[12:23] you do 10 million repetitions over and
[12:26] over again. And this is how Hyperfine is
[12:29] able to determine that there are
[12:31] outliers, there is load. And
[12:33] statistically, it's very important to
[12:35] have multiple runs and then average
[12:38] them.
[12:42] This goes into the sixth uh problem that
[12:44] you can have with benchmarking. Not
[12:47] using statistical methods to compare the
[12:50] performance. And um one problem with
[12:54] benchmarking and comparing stuff against
[12:55] each other is that there is a lot of
[12:58] variance in benchmarks. And with
[13:01] statistical tests, you can give a range
[13:06] uh that is statistically significant for
[13:09] benchmarks that you run. So let's take
[13:11] again the benchmarking loop code that we
[13:14] had before. PHP bench fn optimized
[13:21] away.php.
[13:24] We run both tests against each other
[13:28] and we can see that um Hyperfine shows
[13:34] the range of the results it has and it
[13:38] also um calculates um the variance and
[13:42] it says not that the um the the test ran
[13:45] for 127 milliseconds on average, but it
[13:50] also says that statistically it um
[13:54] fluctuates by 36 milliseconds plus minus
[13:58] around this average. And the same is for
[14:01] the second test. It's 229 milliseconds
[14:05] plus minus 18 milliseconds
[14:07] statistically. So you see this sort of
[14:10] range and then you can compare if both
[14:14] ranges interlap over each other and if
[14:17] the ranges overlap for both tests then
[14:20] this means that statistically they might
[14:23] not actually be different that much
[14:26] because you could see the same
[14:27] performance for both of them.
[14:30] In this case we can see the inter um the
[14:33] ranges don't really overlap. So
[14:35] statistically this is really a different
[14:37] performance between the two scripts.
[14:42] The seventh problem is optimizing what
[14:45] doesn't matter and um this is a problem
[14:48] that uh happens a lot with
[14:50] microbenchmarking.
[14:52] So if you're microbenchmarking some
[14:54] construct against each other and then
[14:57] you're increasing the iterations to 10
[15:00] million, 20 million, 50 million, then
[15:02] you have to ask yourself, are you
[15:04] actually running this that often in your
[15:07] actual PHP code? And then if in the
[15:09] actual PHP code you're only running it
[15:11] like 100 times then maybe even if you
[15:14] can optimize something then it wouldn't
[15:17] matter for the end for your uh script
[15:19] itself because um in real production
[15:22] code it would be just a few nanoseconds
[15:25] uh improvement that isn't really
[15:27] measurable. Let's take for example um
[15:30] this case where we run the difference
[15:33] between uh calling a function a prefix
[15:37] with a name space. So we importing it or
[15:39] we calling it with a prefix name and
[15:41] doing not doing that. So we're not
[15:43] importing that. Um and the difference
[15:46] here is that with this um in the case
[15:49] where it's imported opach can run some
[15:52] uh optimizations and inline the function
[15:55] and in this case here um it cannot do
[15:57] that. So let's run uh both against each
[16:01] other. We can see for the unoptimized
[16:04] code it runs for 500 milliseconds. For
[16:07] the optimized code, it runs for 300
[16:10] something milliseconds and the
[16:12] difference is 1.6 times faster. However,
[16:17] when we look at this, we are really
[16:18] calling count and checking this 20
[16:21] million times. So, the question really
[16:23] is, is this happening 20 million times
[16:26] in our application? Uh, if yes, then
[16:29] this is a very good optimization. uh if
[16:31] count is only called 1,000 times, then
[16:34] you won't see a difference at all. So,
[16:37] let's go back to a full example from
[16:39] Fala's blog post where he tests the uh
[16:43] sprint f changes um that compile down to
[16:47] string concatenation that were added in
[16:50] PHP 8.4 against um each other uh
[16:53] previous versions and using batch
[16:55] classes and not. So, the example first
[16:58] checks for X debug as we've seen. It
[17:00] checks for Zent OPC cache and that it's
[17:02] enabled. Uh has 10 million iterations.
[17:06] Two cases. Um I can run the script with
[17:09] backslash or without backslash. And the
[17:12] functions are declared here as using a
[17:15] backslash. Then this will trigger the
[17:19] optimization in PHP 8.4 or without a
[17:22] backslash it will not trigger the
[17:24] iteration. So we are iterating over um
[17:29] so running 10 million iterations calling
[17:32] this function with a string and an
[17:35] identifier concatenating this.
[17:38] So what Fulka showed is that Hyperfine
[17:41] can be used to parameterize tests in a
[17:46] very interesting way. Um so it allows
[17:49] you to more easily run benchmarks with a
[17:51] lot of different parameters.
[17:54] I have the command here. So we can use
[17:57] the minus L flag to introduce a
[18:00] parameter. In our case, we want to run
[18:03] H.2 against H.4. And then we also
[18:06] introduce a second parameter mode with
[18:09] backslash and without backslash. And
[18:11] then uh we specify only one command PHP
[18:15] and then appending the version number
[18:17] running the bench script that we see
[18:20] here above and then the mode. This will
[18:22] create four tests run against each
[18:24] other. So PHP 8.2 8.2 with backslash
[18:30] runs around 800 milliseconds. PHP 8.4
[18:35] with backslash runs much faster 580
[18:38] milliseconds. And then again PHP 82
[18:42] without a backslash runs similarly
[18:45] around 800 milliseconds. And then u PHP
[18:49] 8.4 4 without a backslash also runs at
[18:52] around 800 milliseconds.
[18:55] And we see the end result. PHP 8.4 um
[18:59] with backslash run 1.3 times faster than
[19:03] 8.2 with a backslash 1.4 times 1.48
[19:08] times faster than without a backslash.
[19:12] So this helps seeing the performance
[19:15] difference. We run the same script and
[19:18] we add some variation and parameters
[19:20] through Hyperfine. This video gave a
[19:22] good overview about seven different
[19:25] problems that you can have uh
[19:26] benchmarking PHP code. Uh how we use the
[19:29] Hyperfine tool, different ways of using
[19:31] Hyperfine and I hope that you really
[19:34] took something away from it for your own
[19:35] benchmarking. If you like this content
[19:38] about PHP performance, you can follow
[19:40] this channel on YouTube or subscribe to
[19:43] our newsletter. The link is also in the
[19:45] description. Thank you very much.