What is Statistical Significance?
In the context of AB testing experiments, statistical significance is how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance.
For example, if you run a test with a 95% significance level, you can be 95% confident that the differences are real.
Calculating statistical significance is quite complex (https://en.wikipedia.org/wiki/Statistical_significance) but the good news is that TubeBuddy does it for you automatically.
Our calculations are focused on Click-Through Rate (CTR). This involves analyzing the number of Impressions each test version received and the corresponding number of clicks. Our analysis does not factor in views, watch time, or any other metrics.
If you'd like to play around with some numbers and see how significance is affected, take a look at https://neilpatel.com/ab-testing-calculator/ and plug in some numbers.
A Simple Example
Let's say you run a test and the original version of your video is shown to 10 people, of which 1 click to watch. Your CTR is 10%. And your variation was shown to 15 people and 2 clicked giving the variation a 12.3% CTR. Can you trust that the variation will always outperform the original? Probably not. They were only shown to 25 people in total.
But let's say you do the same test and the original is shown to 100,000 and of that, 10,000 click - giving it a 10% CTR. And the variation is shown to 150,000 people and 20,000 click giving it a 12.3% CTR. Can you trust these numbers? Yes. They were shown to SO many people that there is enough evidence to show that the variation will perform better in the long run.