[00:00]
Benchmarking PHP code is fun and it has
[00:02]
always fascinated me, but it's also
[00:05]
tricky and there are more ways to do it
[00:07]
wrong than you can count on two hands.
[00:10]
And if you're benchmarking wrong, then
[00:12]
the results are at best useless, at
[00:15]
worst harmful. And at the end of this
[00:18]
video, you have learned a list of
[00:19]
preconditions for good benchmarking in
[00:22]
PHP. One tool that has helped me
[00:24]
immensely in designing good benchmarks
[00:26]
is Hyperfine. And I want to show you how
[00:28]
it works, what its benefits are, and
[00:30]
what problems you still need to account
[00:32]
for. Mo, I am Benjamin and I have helped
[00:36]
thousands of developers with PHP
[00:38]
performance over the last 10 years. What
[00:40]
is Hyperfine? You can write benchmarking
[00:43]
code easily in PHP without additional
[00:46]
help using HR time function or more
[00:50]
commonly seen the micro time function.
[00:53]
After doing this for myself for years,
[00:56]
my colleague Fulker convinced me to give
[00:58]
Hyperfine a try and it stuck. Why even
[01:02]
use a dedicated tool for benchmarking?
[01:04]
Fulker wrote about Hyperfine in our blog
[01:06]
and we are going through his work and is
[01:09]
examples to understand why Hyperfine
[01:11]
helps with avoiding a bunch of
[01:13]
benchmarking problems that are very
[01:15]
common but also introduces some. You can
[01:18]
install uh Hyperfine easily on a Mac
[01:21]
with brew install Hyperfine and on many
[01:24]
other systems as well with a package
[01:26]
manager. And if you have it installed,
[01:28]
you can prefix any command with
[01:30]
Hyperfine. For example, in this case for
[01:32]
the bench sleep script which runs u
[01:36]
sleep um for 100 milliseconds
[01:43]
and then you can also run it uh for
[01:46]
multiple commands. For example, PHP
[01:48]
bench sleep, PHP with a factor of two.
[01:52]
And then again PHP bench sleep
[01:56]
with a factor of three. And then it will
[01:58]
run each of them individually and it
[02:01]
will compare the performance against
[02:03]
each other. This way you can run
[02:05]
alternative implementations for your
[02:07]
code and then benchmark them against
[02:10]
each other and see which one is faster,
[02:11]
which is slower and pick one of um the
[02:15]
solutions that you want to use. Uh we
[02:18]
use it for example also for testing the
[02:21]
performance of different PHP versions
[02:23]
against each other. Let's say if then
[02:25]
improve is there an improvement in a
[02:27]
version and then tests the same code
[02:29]
against multiple versions of PHP and see
[02:32]
how the performance changed. So what
[02:34]
problems do exist with benchmarking? Now
[02:36]
I want to talk about seven different
[02:38]
things that you should account for when
[02:40]
benchmarking PHP code.
[02:44]
The first problem we should look into is
[02:46]
not using a production equivalent
[02:49]
environment. And this can happen because
[02:51]
we are running the benchmarks uh using
[02:53]
the PHP CLI command. This means that by
[02:57]
default opcache is not loaded or not
[03:00]
enabled for the CLI. And this can
[03:03]
drastically change the performance um
[03:06]
compared to running the same code in a
[03:08]
web server. So let's check that Z
[03:11]
opcache is enabled and also in get
[03:18]
opcache enable CLSI is set otherwise
[03:23]
throw new exception
[03:27]
opach not loaded.
[03:31]
So let's run
[03:36]
the code again. we see oh okay opache
[03:39]
was not loaded so let's change
[03:45]
the ini for the cli settings we load
[03:48]
opcache
[03:50]
and next run works
[03:52]
uh what we also should account for is
[03:54]
that xdebug is not loaded so this is
[03:57]
something where I had like several
[04:00]
embarrassing moments where I tested for
[04:03]
example I think PHP P 5.6 against PHP 7
[04:09]
or so and I saw it so much faster PHP 7
[04:14]
and then realized not only of course
[04:17]
PHP7 was much faster than PHP56
[04:20]
but I was off by a big factor because on
[04:23]
PHP 5.6 I also had XD debug running and
[04:27]
I didn't have that on um
[04:32]
I didn't have that on PHP7. So uh if PHP
[04:36]
XD debug uh either either it's not
[04:38]
loaded or throw new exception XD debug
[04:43]
is loaded and then we run that and this
[04:48]
is something that uh specifically with
[04:50]
benchmarking can happen because it's a
[04:52]
system you run them on a system where
[04:54]
you also develop. So it might be the
[04:57]
case that XDbug is actually on. So here
[05:00]
it's off um on my machine and that's
[05:02]
fine.
[05:04]
We could also account for other things
[05:06]
that are non-production ready. It
[05:07]
depends on your environment um what what
[05:10]
you think of what you should account
[05:11]
for.
[05:14]
The second problem that you can have is
[05:16]
writing PHP code that opcache optimizes
[05:19]
away. So let's say we have um code here
[05:24]
that has a function fu returning one and
[05:29]
um we want to test if adding a function
[05:32]
returning this is faster than just
[05:34]
iterating the code here. So this is the
[05:37]
original code that we want to test and
[05:40]
um we are iterating this and dumping at
[05:43]
the end and then we have also the code
[05:46]
where we call the full function and we
[05:49]
compare that against each other. So
[05:51]
let's run PHP
[05:55]
bench
[05:56]
function loop PHP against PHP
[06:01]
bench function
[06:05]
optimized away.php and the the file name
[06:08]
already gives away the problem. Uh
[06:11]
what's happening here is that with
[06:13]
opcache
[06:15]
some of the functions are actually
[06:17]
inlined and optimized away. So because
[06:20]
the fu function is returning a static
[06:22]
value
[06:24]
um it will actually compile the code
[06:26]
down to look exactly like this. Um and
[06:30]
that means that you need to be careful
[06:32]
with certain functions that you don't
[06:35]
write them in a way that opcache
[06:37]
optimizes them for your benchmarking
[06:40]
scenario and in production you're using
[06:43]
a different function. you write it a
[06:45]
little bit differently or it's more
[06:47]
complex because that's just the way
[06:49]
production code is. So um that way you
[06:52]
would see that um the performance is
[06:56]
quite different. So what we found out
[06:58]
here is that actually the code for um
[07:03]
the function optimized away ran faster
[07:06]
than the code um that's just
[07:09]
incrementing the number. So if we relied
[07:11]
on that benchmark, we would think that
[07:13]
functions in PHP code are faster than
[07:16]
just incrementing an integer, which is
[07:18]
not true. Opcimize this away. In this
[07:22]
case, we could fool OPC cache um into
[07:24]
thinking this function cannot um uh be
[07:28]
inlined
[07:30]
by writing some dummy code that it gets
[07:34]
confused by. And if we rerun the
[07:36]
benchmarks now then we see the loop is
[07:39]
at 100 milliseconds and then the
[07:41]
function call is uh much slower. The
[07:44]
factor is it's a factor of two and this
[07:47]
is way more realistic. Now
[07:51]
the third problem is not accounting for
[07:53]
load on the machine and this happens
[07:55]
with developer machines all the time.
[07:58]
So, my developer machine has Docker
[08:00]
running, has Spotify running, has a
[08:03]
browser running, open with 10,000 tabs.
[08:05]
Yes, I'm a tap messy. And it is usually
[08:09]
under quite a bit of load. In this
[08:11]
example, we see what's uh happening
[08:13]
under load. the benchmark can be off
[08:17]
because the um uh the machine actually
[08:21]
doesn't have enough CPU to run a
[08:23]
benchmark um on a CPU core without
[08:27]
getting interrupted. And Hyperfind
[08:29]
checks for interruptions. It checks for
[08:32]
statistical outliers and it warns you
[08:35]
about this. If there are outliers, if
[08:37]
there are problems, it um explains to
[08:40]
you that you should run it on a quieter
[08:42]
system. And this is very helpful because
[08:44]
you can now run the benchmarks as long
[08:48]
as
[08:50]
um
[08:53]
uh you have a run that doesn't show a
[08:55]
warning of this kind. Um maybe you need
[08:58]
to stop a few programs for this to
[09:00]
happen. Um but you will know that like
[09:03]
the benchmarks here are not
[09:04]
statistically valid because um there was
[09:08]
a big like spike or change.
[09:12]
The third problem you can run into when
[09:14]
benchmarking, at least with Hyperfine,
[09:16]
is that you don't account for the
[09:18]
startup time of PHP itself. So let's run
[09:22]
a PHP script through Hyperfine that does
[09:25]
absolutely nothing. So it's PHP
[09:27]
empty.php. We run it and we see that
[09:32]
running this empty script takes 50
[09:34]
milliseconds. And this means that
[09:37]
running any PHP script like with this
[09:40]
PHP um runs 50 milliseconds needs 50
[09:43]
milliseconds of startup time and we need
[09:46]
to discount that from all the benchmarks
[09:48]
that we are running. So maybe you
[09:50]
remember our benchmark from the
[09:51]
beginning with the sleep. We can run it
[09:54]
again. PHP bench sleep.php with a factor
[09:58]
of one. We just had 100 milliseconds of
[10:00]
US sleep.
[10:03]
And as we can see the script is actually
[10:06]
not running 100 milliseconds. On average
[10:08]
it's running 167 milliseconds. And this
[10:11]
is because of the startup time of PHP
[10:14]
itself. And you need to discount that uh
[10:17]
to be able to compare the different runs
[10:19]
with each other. What we usually do is
[10:23]
we um make sure that the benchmark
[10:26]
script uh runs at least 500 milliseconds
[10:29]
or better like around 800 milliseconds.
[10:32]
so that the actual startup time is um
[10:35]
becoming a very small part of this
[10:38]
runtime.
[10:41]
The next problem that you can have with
[10:42]
benchmarking is not repeating the tests
[10:45]
and this is a problem that uh Hyperfind
[10:48]
accounts for automatically. Um as you
[10:51]
remember uh when we run hyperfind for
[10:54]
example on our
[10:58]
um function bench function loop.php.
[11:03]
So what we do here is uh 10 million
[11:08]
iterations of an increment
[11:12]
then it will run this 15 times and um
[11:16]
Hyperfine looks at the execution length
[11:20]
of the first test and then determines
[11:23]
how often it should run them to be able
[11:25]
to make a statistically good um estimate
[11:29]
of the performance. And the reason is
[11:31]
you should account for this is first um
[11:36]
to detect outliers on the machine. So
[11:39]
with this we are able to detect that one
[11:41]
run took significantly uh longer like
[11:44]
this one here. And this might
[11:46]
potentially mean that the machine is
[11:47]
under load and the benchmark is not
[11:49]
reliable. But also you cannot just take
[11:53]
one um
[11:56]
um test and then use this number as the
[12:00]
truth. That is not statistically um good
[12:03]
and you can be way off by not accounting
[12:06]
for changes that um or variances in
[12:09]
this. Maybe you could argue that um
[12:12]
running 10 million iterations is already
[12:14]
repetition, but uh for me it's really
[12:17]
not. The problem is that you want to see
[12:20]
that the performance runs the same when
[12:23]
you do 10 million repetitions over and
[12:26]
over again. And this is how Hyperfine is
[12:29]
able to determine that there are
[12:31]
outliers, there is load. And
[12:33]
statistically, it's very important to
[12:35]
have multiple runs and then average
[12:38]
them.
[12:42]
This goes into the sixth uh problem that
[12:44]
you can have with benchmarking. Not
[12:47]
using statistical methods to compare the
[12:50]
performance. And um one problem with
[12:54]
benchmarking and comparing stuff against
[12:55]
each other is that there is a lot of
[12:58]
variance in benchmarks. And with
[13:01]
statistical tests, you can give a range
[13:06]
uh that is statistically significant for
[13:09]
benchmarks that you run. So let's take
[13:11]
again the benchmarking loop code that we
[13:14]
had before. PHP bench fn optimized
[13:21]
away.php.
[13:24]
We run both tests against each other
[13:28]
and we can see that um Hyperfine shows
[13:34]
the range of the results it has and it
[13:38]
also um calculates um the variance and
[13:42]
it says not that the um the the test ran
[13:45]
for 127 milliseconds on average, but it
[13:50]
also says that statistically it um
[13:54]
fluctuates by 36 milliseconds plus minus
[13:58]
around this average. And the same is for
[14:01]
the second test. It's 229 milliseconds
[14:05]
plus minus 18 milliseconds
[14:07]
statistically. So you see this sort of
[14:10]
range and then you can compare if both
[14:14]
ranges interlap over each other and if
[14:17]
the ranges overlap for both tests then
[14:20]
this means that statistically they might
[14:23]
not actually be different that much
[14:26]
because you could see the same
[14:27]
performance for both of them.
[14:30]
In this case we can see the inter um the
[14:33]
ranges don't really overlap. So
[14:35]
statistically this is really a different
[14:37]
performance between the two scripts.
[14:42]
The seventh problem is optimizing what
[14:45]
doesn't matter and um this is a problem
[14:48]
that uh happens a lot with
[14:50]
microbenchmarking.
[14:52]
So if you're microbenchmarking some
[14:54]
construct against each other and then
[14:57]
you're increasing the iterations to 10
[15:00]
million, 20 million, 50 million, then
[15:02]
you have to ask yourself, are you
[15:04]
actually running this that often in your
[15:07]
actual PHP code? And then if in the
[15:09]
actual PHP code you're only running it
[15:11]
like 100 times then maybe even if you
[15:14]
can optimize something then it wouldn't
[15:17]
matter for the end for your uh script
[15:19]
itself because um in real production
[15:22]
code it would be just a few nanoseconds
[15:25]
uh improvement that isn't really
[15:27]
measurable. Let's take for example um
[15:30]
this case where we run the difference
[15:33]
between uh calling a function a prefix
[15:37]
with a name space. So we importing it or
[15:39]
we calling it with a prefix name and
[15:41]
doing not doing that. So we're not
[15:43]
importing that. Um and the difference
[15:46]
here is that with this um in the case
[15:49]
where it's imported opach can run some
[15:52]
uh optimizations and inline the function
[15:55]
and in this case here um it cannot do
[15:57]
that. So let's run uh both against each
[16:01]
other. We can see for the unoptimized
[16:04]
code it runs for 500 milliseconds. For
[16:07]
the optimized code, it runs for 300
[16:10]
something milliseconds and the
[16:12]
difference is 1.6 times faster. However,
[16:17]
when we look at this, we are really
[16:18]
calling count and checking this 20
[16:21]
million times. So, the question really
[16:23]
is, is this happening 20 million times
[16:26]
in our application? Uh, if yes, then
[16:29]
this is a very good optimization. uh if
[16:31]
count is only called 1,000 times, then
[16:34]
you won't see a difference at all. So,
[16:37]
let's go back to a full example from
[16:39]
Fala's blog post where he tests the uh
[16:43]
sprint f changes um that compile down to
[16:47]
string concatenation that were added in
[16:50]
PHP 8.4 against um each other uh
[16:53]
previous versions and using batch
[16:55]
classes and not. So, the example first
[16:58]
checks for X debug as we've seen. It
[17:00]
checks for Zent OPC cache and that it's
[17:02]
enabled. Uh has 10 million iterations.
[17:06]
Two cases. Um I can run the script with
[17:09]
backslash or without backslash. And the
[17:12]
functions are declared here as using a
[17:15]
backslash. Then this will trigger the
[17:19]
optimization in PHP 8.4 or without a
[17:22]
backslash it will not trigger the
[17:24]
iteration. So we are iterating over um
[17:29]
so running 10 million iterations calling
[17:32]
this function with a string and an
[17:35]
identifier concatenating this.
[17:38]
So what Fulka showed is that Hyperfine
[17:41]
can be used to parameterize tests in a
[17:46]
very interesting way. Um so it allows
[17:49]
you to more easily run benchmarks with a
[17:51]
lot of different parameters.
[17:54]
I have the command here. So we can use
[17:57]
the minus L flag to introduce a
[18:00]
parameter. In our case, we want to run
[18:03]
H.2 against H.4. And then we also
[18:06]
introduce a second parameter mode with
[18:09]
backslash and without backslash. And
[18:11]
then uh we specify only one command PHP
[18:15]
and then appending the version number
[18:17]
running the bench script that we see
[18:20]
here above and then the mode. This will
[18:22]
create four tests run against each
[18:24]
other. So PHP 8.2 8.2 with backslash
[18:30]
runs around 800 milliseconds. PHP 8.4
[18:35]
with backslash runs much faster 580
[18:38]
milliseconds. And then again PHP 82
[18:42]
without a backslash runs similarly
[18:45]
around 800 milliseconds. And then u PHP
[18:49]
8.4 4 without a backslash also runs at
[18:52]
around 800 milliseconds.
[18:55]
And we see the end result. PHP 8.4 um
[18:59]
with backslash run 1.3 times faster than
[19:03]
8.2 with a backslash 1.4 times 1.48
[19:08]
times faster than without a backslash.
[19:12]
So this helps seeing the performance
[19:15]
difference. We run the same script and
[19:18]
we add some variation and parameters
[19:20]
through Hyperfine. This video gave a
[19:22]
good overview about seven different
[19:25]
problems that you can have uh
[19:26]
benchmarking PHP code. Uh how we use the
[19:29]
Hyperfine tool, different ways of using
[19:31]
Hyperfine and I hope that you really
[19:34]
took something away from it for your own
[19:35]
benchmarking. If you like this content
[19:38]
about PHP performance, you can follow
[19:40]
this channel on YouTube or subscribe to
[19:43]
our newsletter. The link is also in the
[19:45]
description. Thank you very much.