---
title: 'Benchmarking PHP code the right way'
source: 'https://youtube.com/watch?v=GkN3pEmMYzU'
video_id: 'GkN3pEmMYzU'
date: 2026-06-14
duration_sec: 1189
---

# Benchmarking PHP code the right way

> Source: [Benchmarking PHP code the right way](https://youtube.com/watch?v=GkN3pEmMYzU)

## Summary

This video discusses the challenges of benchmarking PHP code and introduces Hyperfine, a tool that helps avoid common pitfalls. It covers seven key problems to account for when benchmarking, such as environment differences, opcache optimizations, and machine load, and demonstrates how Hyperfine addresses them.

### Key Points

- **What is Hyperfine?** [00:40] — Hyperfine is a command-line benchmarking tool that can be installed via package managers like brew. It allows running multiple commands and comparing their performance.
- **Problem 1: Not using production-equivalent environment** [02:44] — CLI PHP may not have opcache enabled by default, and xdebug should be disabled. The video shows how to check and enable opcache for CLI.
- **Problem 2: Opcache optimizing away code** [05:14] — Opcache may inline simple functions, making benchmarks misleading. The video demonstrates how to prevent this by adding dummy code.
- **Problem 3: Not accounting for machine load** [07:51] — Developer machines often have background processes that affect benchmarks. Hyperfine detects outliers and warns about load.
- **Problem 4: Not accounting for PHP startup time** [09:12] — PHP has a startup overhead (~50ms). The video recommends making benchmark scripts run at least 500-800ms to minimize its impact.
- **Problem 5: Not repeating tests** [10:41] — Hyperfine automatically runs multiple iterations to detect outliers and provide statistically reliable results.
- **Problem 6: Not using statistical methods** [12:42] — Hyperfine provides mean and standard deviation, allowing comparison of overlapping ranges to determine if differences are significant.
- **Problem 7: Optimizing what doesn't matter** [14:42] — Microbenchmarks may show large improvements that are negligible in real applications if the code is not executed frequently.

### Conclusion

Proper benchmarking requires accounting for environment, opcache, machine load, startup time, repetition, statistics, and relevance. Hyperfine helps with many of these issues, but developers must still ensure their benchmarks reflect real-world usage.

## Transcript

Benchmarking PHP code is fun and it has
always fascinated me, but it's also
tricky and there are more ways to do it
wrong than you can count on two hands.
And if you're benchmarking wrong, then
the results are at best useless, at
worst harmful. And at the end of this
video, you have learned a list of
preconditions for good benchmarking in
PHP. One tool that has helped me
immensely in designing good benchmarks
is Hyperfine. And I want to show you how
it works, what its benefits are, and
what problems you still need to account
for. Mo, I am Benjamin and I have helped
thousands of developers with PHP
performance over the last 10 years. What
is Hyperfine? You can write benchmarking
code easily in PHP without additional
help using HR time function or more
commonly seen the micro time function.
After doing this for myself for years,
my colleague Fulker convinced me to give
Hyperfine a try and it stuck. Why even
use a dedicated tool for benchmarking?
Fulker wrote about Hyperfine in our blog
and we are going through his work and is
examples to understand why Hyperfine
helps with avoiding a bunch of
benchmarking problems that are very
common but also introduces some. You can
install uh Hyperfine easily on a Mac
with brew install Hyperfine and on many
other systems as well with a package
manager. And if you have it installed,
you can prefix any command with
Hyperfine. For example, in this case for
the bench sleep script which runs u
sleep um for 100 milliseconds
and then you can also run it uh for
multiple commands. For example, PHP
bench sleep, PHP with a factor of two.
And then again PHP bench sleep
with a factor of three. And then it will
run each of them individually and it
will compare the performance against
each other. This way you can run
alternative implementations for your
code and then benchmark them against
each other and see which one is faster,
which is slower and pick one of um the
solutions that you want to use. Uh we
use it for example also for testing the
performance of different PHP versions
against each other. Let's say if then
improve is there an improvement in a
version and then tests the same code
against multiple versions of PHP and see
how the performance changed. So what
problems do exist with benchmarking? Now
I want to talk about seven different
things that you should account for when
benchmarking PHP code.
The first problem we should look into is
not using a production equivalent
environment. And this can happen because
we are running the benchmarks uh using
the PHP CLI command. This means that by
default opcache is not loaded or not
enabled for the CLI. And this can
drastically change the performance um
compared to running the same code in a
web server. So let's check that Z
opcache is enabled and also in get
opcache enable CLSI is set otherwise
throw new exception
opach not loaded.
So let's run
the code again. we see oh okay opache
was not loaded so let's change
the ini for the cli settings we load
opcache
and next run works
uh what we also should account for is
that xdebug is not loaded so this is
something where I had like several
embarrassing moments where I tested for
example I think PHP P 5.6 against PHP 7
or so and I saw it so much faster PHP 7
and then realized not only of course
PHP7 was much faster than PHP56
but I was off by a big factor because on
PHP 5.6 I also had XD debug running and
I didn't have that on um
I didn't have that on PHP7. So uh if PHP
XD debug uh either either it's not
loaded or throw new exception XD debug
is loaded and then we run that and this
is something that uh specifically with
benchmarking can happen because it's a
system you run them on a system where
you also develop. So it might be the
case that XDbug is actually on. So here
it's off um on my machine and that's
fine.
We could also account for other things
that are non-production ready. It
depends on your environment um what what
you think of what you should account
for.
The second problem that you can have is
writing PHP code that opcache optimizes
away. So let's say we have um code here
that has a function fu returning one and
um we want to test if adding a function
returning this is faster than just
iterating the code here. So this is the
original code that we want to test and
um we are iterating this and dumping at
the end and then we have also the code
where we call the full function and we
compare that against each other. So
let's run PHP
bench
function loop PHP against PHP
bench function
optimized away.php and the the file name
already gives away the problem. Uh
what's happening here is that with
opcache
some of the functions are actually
inlined and optimized away. So because
the fu function is returning a static
value
um it will actually compile the code
down to look exactly like this. Um and
that means that you need to be careful
with certain functions that you don't
write them in a way that opcache
optimizes them for your benchmarking
scenario and in production you're using
a different function. you write it a
little bit differently or it's more
complex because that's just the way
production code is. So um that way you
would see that um the performance is
quite different. So what we found out
here is that actually the code for um
the function optimized away ran faster
than the code um that's just
incrementing the number. So if we relied
on that benchmark, we would think that
functions in PHP code are faster than
just incrementing an integer, which is
not true. Opcimize this away. In this
case, we could fool OPC cache um into
thinking this function cannot um uh be
inlined
by writing some dummy code that it gets
confused by. And if we rerun the
benchmarks now then we see the loop is
at 100 milliseconds and then the
function call is uh much slower. The
factor is it's a factor of two and this
is way more realistic. Now
the third problem is not accounting for
load on the machine and this happens
with developer machines all the time.
So, my developer machine has Docker
running, has Spotify running, has a
browser running, open with 10,000 tabs.
Yes, I'm a tap messy. And it is usually
under quite a bit of load. In this
example, we see what's uh happening
under load. the benchmark can be off
because the um uh the machine actually
doesn't have enough CPU to run a
benchmark um on a CPU core without
getting interrupted. And Hyperfind
checks for interruptions. It checks for
statistical outliers and it warns you
about this. If there are outliers, if
there are problems, it um explains to
you that you should run it on a quieter
system. And this is very helpful because
you can now run the benchmarks as long
as
um
uh you have a run that doesn't show a
warning of this kind. Um maybe you need
to stop a few programs for this to
happen. Um but you will know that like
the benchmarks here are not
statistically valid because um there was
a big like spike or change.
The third problem you can run into when
benchmarking, at least with Hyperfine,
is that you don't account for the
startup time of PHP itself. So let's run
a PHP script through Hyperfine that does
absolutely nothing. So it's PHP
empty.php. We run it and we see that
running this empty script takes 50
milliseconds. And this means that
running any PHP script like with this
PHP um runs 50 milliseconds needs 50
milliseconds of startup time and we need
to discount that from all the benchmarks
that we are running. So maybe you
remember our benchmark from the
beginning with the sleep. We can run it
again. PHP bench sleep.php with a factor
of one. We just had 100 milliseconds of
US sleep.
And as we can see the script is actually
not running 100 milliseconds. On average
it's running 167 milliseconds. And this
is because of the startup time of PHP
itself. And you need to discount that uh
to be able to compare the different runs
with each other. What we usually do is
we um make sure that the benchmark
script uh runs at least 500 milliseconds
or better like around 800 milliseconds.
so that the actual startup time is um
becoming a very small part of this
runtime.
The next problem that you can have with
benchmarking is not repeating the tests
and this is a problem that uh Hyperfind
accounts for automatically. Um as you
remember uh when we run hyperfind for
example on our
um function bench function loop.php.
So what we do here is uh 10 million
iterations of an increment
then it will run this 15 times and um
Hyperfine looks at the execution length
of the first test and then determines
how often it should run them to be able
to make a statistically good um estimate
of the performance. And the reason is
you should account for this is first um
to detect outliers on the machine. So
with this we are able to detect that one
run took significantly uh longer like
this one here. And this might
potentially mean that the machine is
under load and the benchmark is not
reliable. But also you cannot just take
one um
um test and then use this number as the
truth. That is not statistically um good
and you can be way off by not accounting
for changes that um or variances in
this. Maybe you could argue that um
running 10 million iterations is already
repetition, but uh for me it's really
not. The problem is that you want to see
that the performance runs the same when
you do 10 million repetitions over and
over again. And this is how Hyperfine is
able to determine that there are
outliers, there is load. And
statistically, it's very important to
have multiple runs and then average
them.
This goes into the sixth uh problem that
you can have with benchmarking. Not
using statistical methods to compare the
performance. And um one problem with
benchmarking and comparing stuff against
each other is that there is a lot of
variance in benchmarks. And with
statistical tests, you can give a range
uh that is statistically significant for
benchmarks that you run. So let's take
again the benchmarking loop code that we
had before. PHP bench fn optimized
away.php.
We run both tests against each other
and we can see that um Hyperfine shows
the range of the results it has and it
also um calculates um the variance and
it says not that the um the the test ran
for 127 milliseconds on average, but it
also says that statistically it um
fluctuates by 36 milliseconds plus minus
around this average. And the same is for
the second test. It's 229 milliseconds
plus minus 18 milliseconds
statistically. So you see this sort of
range and then you can compare if both
ranges interlap over each other and if
the ranges overlap for both tests then
this means that statistically they might
not actually be different that much
because you could see the same
performance for both of them.
In this case we can see the inter um the
ranges don't really overlap. So
statistically this is really a different
performance between the two scripts.
The seventh problem is optimizing what
doesn't matter and um this is a problem
that uh happens a lot with
microbenchmarking.
So if you're microbenchmarking some
construct against each other and then
you're increasing the iterations to 10
million, 20 million, 50 million, then
you have to ask yourself, are you
actually running this that often in your
actual PHP code? And then if in the
actual PHP code you're only running it
like 100 times then maybe even if you
can optimize something then it wouldn't
matter for the end for your uh script
itself because um in real production
code it would be just a few nanoseconds
uh improvement that isn't really
measurable. Let's take for example um
this case where we run the difference
between uh calling a function a prefix
with a name space. So we importing it or
we calling it with a prefix name and
doing not doing that. So we're not
importing that. Um and the difference
here is that with this um in the case
where it's imported opach can run some
uh optimizations and inline the function
and in this case here um it cannot do
that. So let's run uh both against each
other. We can see for the unoptimized
code it runs for 500 milliseconds. For
the optimized code, it runs for 300
something milliseconds and the
difference is 1.6 times faster. However,
when we look at this, we are really
calling count and checking this 20
million times. So, the question really
is, is this happening 20 million times
in our application? Uh, if yes, then
this is a very good optimization. uh if
count is only called 1,000 times, then
you won't see a difference at all. So,
let's go back to a full example from
Fala's blog post where he tests the uh
sprint f changes um that compile down to
string concatenation that were added in
PHP 8.4 against um each other uh
previous versions and using batch
classes and not. So, the example first
checks for X debug as we've seen. It
checks for Zent OPC cache and that it's
enabled. Uh has 10 million iterations.
Two cases. Um I can run the script with
backslash or without backslash. And the
functions are declared here as using a
backslash. Then this will trigger the
optimization in PHP 8.4 or without a
backslash it will not trigger the
iteration. So we are iterating over um
so running 10 million iterations calling
this function with a string and an
identifier concatenating this.
So what Fulka showed is that Hyperfine
can be used to parameterize tests in a
very interesting way. Um so it allows
you to more easily run benchmarks with a
lot of different parameters.
I have the command here. So we can use
the minus L flag to introduce a
parameter. In our case, we want to run
H.2 against H.4. And then we also
introduce a second parameter mode with
backslash and without backslash. And
then uh we specify only one command PHP
and then appending the version number
running the bench script that we see
here above and then the mode. This will
create four tests run against each
other. So PHP 8.2 8.2 with backslash
runs around 800 milliseconds. PHP 8.4
with backslash runs much faster 580
milliseconds. And then again PHP 82
without a backslash runs similarly
around 800 milliseconds. And then u PHP
8.4 4 without a backslash also runs at
around 800 milliseconds.
And we see the end result. PHP 8.4 um
with backslash run 1.3 times faster than
8.2 with a backslash 1.4 times 1.48
times faster than without a backslash.
So this helps seeing the performance
difference. We run the same script and
we add some variation and parameters
through Hyperfine. This video gave a
good overview about seven different
problems that you can have uh
benchmarking PHP code. Uh how we use the
Hyperfine tool, different ways of using
Hyperfine and I hope that you really
took something away from it for your own
benchmarking. If you like this content
about PHP performance, you can follow
this channel on YouTube or subscribe to
our newsletter. The link is also in the
description. Thank you very much.