I tried YouTube A/B Thumbnail Testing in 8 videos and found BIG problems

[00:00] hello hello hello hope you're doing well

[00:02] all right YouTube thumbnail testing or

[00:04] AB testing as some would call it it has

[00:07] been out now for a couple of months and

[00:09] I have had the chance to do some testing

[00:11] in eight of my videos and my honest

[00:14] opinion here is it is good at least

[00:17] initially until I realized that it's

[00:20] horrible the way YouTube implemented

[00:22] this let me start by asking you a simple

[00:25] question what exactly is thumbnail

[00:27] testing to you it is basically comparing

[00:29] two or three thumbnails than checking to

[00:32] see which one has a higher click through

[00:33] itate am I right that was my initial

[00:36] thought and I instinctively think that

[00:38] that is what it is for so when I did my

[00:40] testing I looked at test results see

[00:43] which has the higher percentage then

[00:45] select that as the winner that is how

[00:48] the feature is meant to help I give

[00:50] YouTube Three thumbnails they test it

[00:52] out for me by swapping at random times

[00:55] then run the test for me if you have

[00:57] been using Tu buddies AB thumbnail test

[01:00] that is how it works as well in fact

[01:02] that is what anyone with common sense

[01:05] would think when we say AB testing ask

[01:07] any U iux designer and they will tell

[01:10] you that too but YouTube for whatever

[01:13] reason decided that the thumbnail should

[01:16] measure watch duration instead of Click

[01:18] through rate I mean why how on Earth can

[01:22] a thumbnail decide someone's watch

[01:24] duration so this is what YouTube has to

[01:26] say how is the winning thumbnail

[01:28] selected the winning thumbnail is

[01:30] selected based on the highest share of a

[01:32] video's watch time but why is watchtime

[01:35] share used great thumbnails don't just

[01:38] get viewers to click they also help

[01:40] viewers understand what the video is

[01:42] about so that they can make informed

[01:45] decision of what to watch all right fair

[01:47] enough by selecting based on watchtime

[01:49] share the winning thumbnail is the one

[01:52] that gets viewers to both click and stay

[01:55] to watch now that doesn't make any sense

[01:57] at all people click on a video because

[01:59] the thumbnail or the title made them

[02:01] curious but what makes them stay to

[02:04] watch the video is about the

[02:06] presentation the flow of the video the

[02:09] storytelling the engagement so what I'm

[02:12] seeing here is even if there's a channel

[02:14] out there who is in the same niche as I

[02:17] am who creates similar thumbnails as me

[02:20] who has the same type of information in

[02:23] our videos we may get the same

[02:25] click-through rate but that does not

[02:27] mean we will get the same watch through

[02:30] ation I'll show you imagine if I'm

[02:31] sharing a tip to making more money on

[02:33] YouTube so yeah guys today I made $100

[02:37] on YouTube and what I did is to make

[02:41] sure I create my video then I upload it

[02:44] and then I share it on other social

[02:46] media platforms and when my video gains

[02:50] Traction in other social media platforms

[02:53] they ended up clicking on my chat link

[02:55] and watching my video on YouTube then

[02:58] getting me more views

[03:00] so I made $100 like that extremely dull

[03:04] not much hand gestures no soul no

[03:07] expression whatsoever now compare that

[03:11] to this so today I made $100 on YouTube

[03:14] and I cannot believe what I did actually

[03:17] works what I did is to make sure I

[03:19] create my video then upload that video

[03:22] to YouTube and then I share it to other

[03:25] social media platforms these social

[03:27] media platforms will then get views on

[03:29] their own attracts new audiences for me

[03:32] and get those who are interested to

[03:34] watch my YouTube video via the link I

[03:36] shared and when my video gain Traction

[03:39] in other social media platforms they

[03:41] ended up clicking on my shared link and

[03:43] watching my video on YouTube than

[03:46] getting me more views on YouTube and

[03:48] increasing my revenue for that video it

[03:51] is as simple as that I didn't believe

[03:53] that would work and I hope you can try

[03:55] this out and see the traction as well

[03:57] same message but the watch time varies

[04:00] depending on the delivery right so I

[04:03] honestly don't think YouTube got this

[04:05] right at all the thumbnail has got

[04:07] nothing to do with watch duration even

[04:10] if the Creator nailed the thumbnail and

[04:12] delivered the promised value but YouTube

[04:15] did not end their explanation there they

[04:17] have more to say they state that

[04:19] additionally YouTube's search and

[04:22] Discovery systems consider both when

[04:24] ranking videos choosing thumbnails based

[04:27] on watchtime share will benefit both you

[04:30] and your viewers the way I read that

[04:32] sentence it is just covering up

[04:34] something they don't understand I feel

[04:36] like the team who came up with this

[04:38] feature needs to be content creator

[04:40] themselves hence I don't buy what they

[04:42] say it isn't benefiting me nor my

[04:45] viewers because the metrics are

[04:47] unrelated having said that I do find one

[04:50] very interesting fact about this and

[04:52] that is how YouTube does rank our video

[04:56] this is something that I have had

[04:57] suspicion on for years that's I think

[05:00] that some videos are ranked much like

[05:03] the Google algorithm and I strongly

[05:05] believe that big channels that have

[05:07] created one or more successful videos

[05:10] their channels are ranked secretly so

[05:13] say if I were to create a video of the

[05:15] same topic as Nate black then my video

[05:18] will not get as much impression as his

[05:20] because his channel is ranked higher

[05:22] than mine and therefore the videos he

[05:25] creates will have the advantage of

[05:26] getting ranked higher of course this is

[05:28] just my specul but that particular

[05:31] sentence from YouTube about ranking

[05:33] videos it kind of tells me that it does

[05:36] exist and is hidden from us anyways

[05:38] that's what I found from YouTube

[05:40] thumbnail testing and set to say I

[05:42] didn't even notice it until recently but

[05:45] not only that another big problem with

[05:47] YouTube thumbnail testing is it doesn't

[05:50] benefit small channels in order to have

[05:53] thumbnail testing perform efficiently

[05:55] you actually need a big sample siiz data

[05:58] for example most of my thumbnail testing

[06:00] takes at least two weeks to conclude and

[06:03] often it ends with no conclusion so

[06:06] that's a big waste of time but that's

[06:07] because my sample size is small meaning

[06:10] my videos aren't getting enough

[06:12] impressions for YouTube to do

[06:13] substantial thumbnail testing and who is

[06:16] responsible for giving me more

[06:18] impressions YouTube of course and here's

[06:20] the thing if you're a small Channel and

[06:23] you don't get as many Impressions

[06:25] because your channel isn't ranked yet

[06:28] then this whole thumbnail testing is

[06:31] useless to you your sample size is just

[06:33] too small to make any proper conclusion

[06:35] which is very counterintuitive because I

[06:38] thought thumbnail testing is for small

[06:40] channels but turns out it is really for

[06:42] the big players again I know shocking

[06:46] right I don't know why but when I first

[06:47] heard about thumbnail testing from

[06:49] YouTube from the big content creators

[06:51] out there who got invited to test this

[06:54] feature like Mr Beast I was given the

[06:57] impression that they are to measure

[06:59] click rate and when I look back at what

[07:01] Mr Beast posted when he first started

[07:03] testing this I think even he got

[07:05] confused about what YouTube's AB

[07:08] thumbnail testing is all about the post

[07:11] he made on Twitter or X or whatever he

[07:14] is clearly talking about click through

[07:17] rate and not watch share look that being

[07:20] said I do still use it these days for

[07:23] every video I launch I create three

[07:26] thumbnail versions and I upload them for

[07:28] testing this three thumbnails contain

[07:31] very subtle changes like font color or

[07:34] face expression but that's about it then

[07:38] I will test it out and 99% of the time I

[07:41] will get almost equal percentages which

[07:44] means it doesn't matter which thumbnail

[07:45] I pick they all give similar results

[07:47] there is one time where YouTube gave me

[07:49] a winning thumbnail after testing just

[07:51] one day but that happen only once and I

[07:55] actually don't know why just take a look

[07:57] yourself they're all the same nail I

[08:00] just tilt one and turn another one

[08:02] upside down I guess it does increase

[08:04] click through rate but again I don't see

[08:07] how that will affect watch time unless

[08:10] of course the formula is as simple as

[08:12] taking the clickthrough rate plus the

[08:15] watch time then giving a final score to

[08:18] that thumbnail to determine the winner

[08:20] so that means assuming watch time stays

[08:22] constants because the video style is

[08:24] only capable of doing so much then the

[08:27] deciding factor of a winning thumbnail

[08:29] automatically becomes the click through

[08:32] rate so I guess in a way it is still the

[08:35] click through rate and we're back to the

[08:38] correct way of AB testing look you can

[08:40] see how confusing this is and YouTube

[08:42] needs to be a little more transparent

[08:44] and be straightforward maybe for once

[08:46] stop working with big channels who have

[08:48] the edits advantage and work with small

[08:51] channels who really want to see if new

[08:54] features introduced can indeed help

[08:57] channels grow big channels are all

[08:59] already established they don't need

[09:01] these tools but small channels do so the

[09:04] very least you can do YouTube is to

[09:07] spend those billions of dollars you have

[09:10] to implement a feature that is made

[09:13] specifically for small channels big

[09:15] channels can help sustain YouTube they

[09:17] don't need as much help small channels

[09:20] though if you help us grow our channel

[09:22] to become the next big Channel then that

[09:26] benefits YouTube in the long run because

[09:28] the pool of big channels just gets

[09:31] bigger it's a win-win situation if you

[09:33] ask me