[00:00] I got my hands on the new Apple Watch Series 12 heart rate tracking study and Apple claims that the new Apple Watch Series 12 and Ultra 4 have the most accurate heart rate sensing of any wearable
[00:12] from any brand. But is that actually true? What do the results show? And most importantly, what do they not show? And we can even see which competitive Apple things are best and which are worse. And you
[00:24] might be surprised. Right here we have the main results for different exercises, but first we need to understand how this study was performed to grasp all the results. This was a pretty large
[00:36] study. Apple enrolled 1460 participants with 1254 contributing data to the final analysis, so just over 1200 people. Participants were both an Apple Watch Series 12 on one wrist and then a
[00:51] competing wearable on the other and as a reference the Polar H10 EZG chest strap was used and both the Apple watch and the competitor were randomly placed on the left and right wrist so the Apple
[01:04] watch should be roughly 50% of the time on the dominant hand and 50% on the non-dominant hand. This is actually quite important and this whole test setup is actually fundamentally similar to how I test heart rate tracking accuracy because I also compare the wearable directly against an
[01:21] ECG chest strap and even the same exact ECG chest strap the Polar H10. By the way for those of you that are new to the channel My name is Rob and I'm a postdoctoral scientist specializing in biological data analysis Which is why I find this study so interesting. Now Apple compared the series 12 against several competitors
[01:39] specifically the Garmin Forerunner 970, the Pixel Watch 4, the Huawei Watch 5, the Galaxy Watch 8, the the Whoop Strap 5.0 and the Oura Ring 5, and participants completed different exercises,
[01:52] which included outdoor running, treadmill intervals, cycling, HIIT exercises, and strength training. And I think these were pretty good choices, because they do include many of the conditions that are challenging for optical heart rate sensors such as cadence loss during running but also gripping during cycling and weightlifting and also rapid heart rate changes during intervals In addition they also tested background heart rate measurements during rest desk work meal prep and walking
[02:20] So a lot of data. I think the exercise part is the most interesting, so I'll be focusing on that. But I'll also quickly touch upon the more resting heart rate. Now Apple primarily evaluated accuracy using two metrics, so the mean absolute error and also the root mean square error.
[02:36] Now mean absolute error calculates how far each watch measurement is from the ECG reference, ignores whether that was too high or too low and then averages that error. And the RMSE is actually very similar, it just gives a bit more weight to larger errors.
[02:52] In the end the results are basically the same no matter which metric you use. So I will use RMSE just because that's the first metric they show, so somehow there's more emphasis on that. And this differs a bit from how I do correlation testing in my reviews.
[03:05] Correlation measures how closely the shape of the watch signal follows the ECG signal. But I basically expect very similar results. But let's just dive right in and let's have a look at how they visualize the data here and what it means.
[03:18] So Apple doesn't actually put the accuracy of the Apple Watch and the accuracy of the other device that was worn. They subtract the two from each other. So that means if the value is left from zero, the Apple Watch did better.
[03:32] And if the value is right from zero, the other devices did better. And if we take a look at sort of the average results over all the participants, so these are these dots right here, always the Apple Watch was doing better than devices,
[03:47] except for maybe in this case where there wasn't a significant difference between the Pixel Watch 4 and the Apple Watch Series 12, and this was for outdoor cycling. But what I found very interesting is then ranking the remaining devices.
[04:01] So if we look at running, the second best watch after the Apple Watch is the Pixel Watch 4, followed by the Huawei Watch 5, and then the Garmin 970. If we take a look at indoor running, it's again the Pixel Watch 4, second best, Huawei Watch 5,
[04:16] and in this case the Wolfstrap. And the Pixel Watch and the Huawei Watch are basically always number 2 and 3 after the Apple Watch And the reason I find it interesting because it true for all different exercises is because these are more smart watch companies and less sport companies
[04:33] And I think this shows how important machine learning has gotten into doing good heart rate measurements. And these sort of older sports companies still have some catching up to do with the companies that have invested a lot in machine learning and AI.
[04:46] and they're using that AI to get better heart rate tracking. I know that for a fact for the Pixel Watch 4 and also from what I read from Apple, they also use some kind of AI to get better heart rate tracking. So even Apple admits here, even though they say they're best,
[05:00] that the Pixel Watch is second best followed by the Huawei Watch and usually the number four is either the Wolfstrap or the Garmin 49970. And those results are pretty consistent.
[05:13] So that's for both outdoor running, indoor running, cycling, HIIT and strength training. You can see that for strength training, the Apple Watch seems to be doing significantly better. So they don't really go into dropout rates as far as I know.
[05:27] So that's something I would be really interested to see. So does the Apple Watch just stop the testing in those moments? Or is there some punishment for leaving out some measurements? That I'm not too sure of. That wasn't too clear for me based on these results. Now Apple also right here summarizes all these results.
[05:42] And if we then rank the different devices, the Apple Watch is number one, the Pixel Watch is number two, the Huawei Watch 5 is number three. This is closely followed by, at number four, the Garmin 40970, then the WhoopStrap 5.0, then the Galaxy Watch.
[05:58] And as expected, the Aura Ring is worse, but the ring is also just not a good form factor for heart rate measurements. And interestingly, if we take a look at the daily living measurements, So not starting an exercise and just the continuous measurements over time.
[06:13] Again, number two is Pixel Watch. Number three is the Wild Watch 5. And of course, in their testing, the Apple Watch is number one. Now, I have to admit here that the Garmin 970 was also very close.
[06:25] And actually, the Wolf Shredder is quite a bit worse here. Now, I think these results are pretty clear. Basically in all cases the Apple Watch is doing best or equally well as the competitors Only for the daily living were there a few occasions like right here where the Pixel Watch actually did
[06:43] better than the Apple Watch and that was also the case here at least on average or here they're actually more or less identical. As far as I can judge based on the method step for use it was a very in-depth study and I do trust that in the way they calculated it the Apple Watch was doing best
[07:00] I do have some questions still about these calculations the main thing being dropouts so at least with the previous heart rate algorithm if the signal got bad the Apple Watch was aware of this and then dropped out these values that is not how typically other wearables do it so it
[07:16] might be that the Apple Watch here has an advantage in that way if it does drop out the bad quality signals and the others don't. I'm not sure if that's the case here and of course Apple has some advantage in getting better results so it's not an independent study but it does show that they put
[07:32] in the effort in both training the model and validating it and I'm actually happily surprised that they didn't shy back from showing that their competitor the Pixel Watch is actually also doing quite well and it's consistently the number two. But of course we always need to take these results
[07:48] with a grain of salt just because Apple did a study that shows that Apple is the best. But again, I do think they put a lot of effort into it. I do have a few more comments about that testing setup or questions that I would have.
[08:00] One is that they use Bluetooth broadcasting for some of the other devices to get the signal. And I don't know if this Bluetooth signal also gives a timestamp with the actual measurements or if that timestamp has just created the moment that it's received and there might
[08:15] some delay which could have some influence on the accuracy but this is something I still want to ask Apple but now it's up to us the reviewers to check if this is also the case for us. This will be at a smaller scale but it will be independent so I think there's a benefit to both. Now if you like this
[08:31] video don't forget to like and subscribe and given that you watched this video you might want to look at this video where I discuss all the other new features of the new Apple Watch Series 12 and the AppleX Ultra 4 and this video on my favorite sleep improvement device, the HP Pod.