I recently tested four different privacy policies on a sign-up form on the home page of a betting community. The results were quite surprising as the variations had a drastically different impact on sign-ups – from an 18.70% drop in sign-ups to an increase of 19.47%.
In this article, I’ll show all four variations, run you through the test data, and give you concrete takeaways.
100% privacy – we will never spam you
Test data: I ran the test for 9 days, reached a sample size of 16152 visits and 297 conversions, and the control variant outperformed the treatment throughout the entire test period. The statistical confidence level was 96%, and the standard error was 0%.
The development of experiment one over the 9-day test period.
Takeaways from experiment 1:
My hypothesis is that – although the messaging revolves around assuring prospects that they won’t be spammed – the word spam itself give rise to anxiety in the mind of the prospects. Therefore, the word should be avoided in close proximity to the form.
100% privacy. We keep all your personal information secret
Test data: I ran this test for 12 days, and after 15675 visitors and 279 conversions, there really was no significant difference between the control variant and the treatment. The treatment started out performing better but tanked as the sample size grew over the 12 day test period.
The treatment started out performing better but tanked as the sample size grew over the 12-day test period.
Takeaways from experiment 2:
The treatment in experiment 2 performed way better than the treatment from experiment 1, but it still didn’t improve conversion.
It would seem that removing the word spam and focusing on the aspect of keeping information secret had a positive effect. Nevertheless, this is still a vague policy that doesn’t say a whole lot.
Ok, so experiment 1 and 2 gave me some valuable insights and helped me develop an idea for yet another treatment. I was still pretty surprised by the results of the first two tests.
For the third treatment, I decided to go for a more authoritative and solid policy with no “cuteness factor”. Moreover, I wanted this treatment to be much clearer than the two previous ones. With all this in mind, I came up with:
We guarantee 100% privacy. Your information will not be shared
Bingo – this treatment was a home run! It increased conversions significantly and gave us 19.47% more sign-ups.
Test data: I ran the test for 12 days and reached a sample size of 20257 visitors and 380 conversions. The statistical confidence level was 96% and the standard error was 0%. From very early on in the test, the treatment outperformed the control variant.
The development of experiment 3 over the 12-day test period.
Takeaways from experiment 3:
The most noticeable change in treatment 3 is the guarantee. In experiment 1 and 2, the policy only said, 100% privacy, whereas the policy in experiment 3 said, We guarantee 100% privacy.
Personally, I find this wording much more credible. Moreover, the second part, Your information will not be shared, is way more clear and authoritative.
Now I was curious to see what would happen if I combined the “best” and “worst” from the previous experiments into one variant – which resulted in treatment 4:
We guarantee 100% privacy. We will never spam you!
Test data: I ran this test for 15 days and after 18959 visitors and 370 conversions, there was no significant difference between the control variant and the treatment policy.
Takeaways from experiment 4:
Treatment 4 was a combination of the best performing variant, We guarantee 100% privacy, and the worst performing variant, We will never spam you.
As the test data revealed that there was no significant difference between the control and the treatment, it would seem that the “good” and the “bad” part parts canceled each other out and therefore had no real impact on sign-ups.
My research points to the fact that a credible, clear policy with a guarantee effectively assures prospects that it’s safe to fill out the form. Moreover, my data suggests that you should be careful with using the word spam – even if the intention is to guarantee against it – as it can backfire and create a higher level of anxiety.
The policy that did best in this series of tests was:
We guarantee 100% privacy. Your information will not be shared.
I’m currently testing it on my newsletter from here on ContentVerve.com, and at the moment the treatment with the policy is performing 6% better than the control – however, the data isn’t conclusive yet.
By the way – This wasn’t the first thing I tested on BettingExpert.com
I’ve done a bunch of other tests on BettingExpert.com. Among other things, I tested a few simple copy tweaks that resulted in an increase in sign-ups of 31.54%.