5d ago in Marketing Tips
Ignore your split tests for the first few days.
Want to improve your conversion rate? Ignore your split tests for the first few days.
The first few days of a split test are VERY dangerous.
Check out the day by day of the test below.
You’ll notice over the first ~4 days the variant was outpacing the control by a large margin… then it flipped.
If you would have taken 4 days worth of data and implemented the test, you would have implemented a worse converting.
So… why does this happen in the first place?
Random Variance is a major culprit. Flip a coin 3x, and you have a 1 in 8 chance of getting Heads three times.
Flip that same coin 10 times and you have a 1/1024 chance of getting heads 10 times in a row.
So… the first few days are just a small amount of data. the longer you run a test, the more data you get, and the less chance what you’re seeing is due to random variation.
For those coming into split testing with a background in paid traffic, this is pretty much the opposite of what happens on ad platforms.
Usually, if an ad works, it works immediately. Meta is TRYING to find buyers. Split testing is completely agnostic to who buys.
Next reason you can’t trust the first few days of split testing data is novelty bias.
If a large percentage of traffic to your site are returning visitors to your site, the control version of your site is super familiar to them.
When you introduce a new element to the site, they’ll pay more attention simply because it’s new.
When you’re analyzing your split test results, this is exactly why you have to analyze your returning visitors separately from new visitors.
I’m all for getting excited over new split test results, but remember, my first rule of CRO is “do no harm”. And 2 or 3 days of data simply is not enough.
0
0 comments
Ignore your split tests for the first few days.
skool.com/lordsofmarketing
Welcome to the house of marketing lords.
Leaderboard (30-day)
1
+6
2
+2
3
+1
4
+1
5
+1
powered by