Is Multivariate Testing Worth Your Time and Money?

A/B testing is a straightforward way to test specific elements of your website: do visitors prefer the red banner, or does the blue work better? Do the text-heavy pages have a higher bounce rate? Do
your calls to action work to get visitors to click through to the next page? This can be a great way to optimise your conversion rates, but then you may start thinking about multiple factors, and how they interact with each other. This is more involved, obviously – but is it worth your time and resources?

Yes, But.

Apologies if you hate ambiguous answers, but yes, it could be worth your time, or maybe not so much. The benefits of doing so are that you can focus in on specific page design elements to see how they affect your conversion rate, and how these elements interact and tweak combinations for CRO.

But, before you launch your test, consider this: there are so many more combinations. Instead of testing a blue banner against a red banner, you’re testing a blue banner with yellow Calibri font with this image with this text… It can certainly become more difficult to see what’s doing what. At the same time, the test setups are restrictive and you cannot place items where you would necessarily want. Another downside is that you need a great deal more traffic to get statistically meaningful data.

There are software programs that you can purchase so this becomes a much easier process; these usually have self- and/or full service help for you, and you can choose a program that does what you need it to, whether it is segmenting traffic or allowing multiple goals. The downside is that these often require an initial startup fee and ongoing licensing costs. Many times this is “a four figure monthly fee,” which can be prohibitive for small and/or local companies.

One more caution before you tackle multivariate testing: split testing (as both A/B and MTV are) requires multiple versions of the same site to be up and running. Random visitors are directed to different versions of the site; normally, this would be considered cloaking. If you do choose to do MTV, make sure you follow Google’s guidelines:

  • Your variations should uphold the ‘spirit’ of your original page’s content — they shouldn’t change its meaning or people’s general perception of it.
  • Keep your source code updated. An experiment should be ended when sufficient data has been collected, and the original page source updated accordingly.
  • Your original page should always be shown to a non-trivial percentage of your users. It’s okay to use Website Optimiser to briefly run your winning combination after stopping an experiment, but be sure to update your test page’s source code to reflect that winning combination soon afterwards.

If you have the traffic and the technology budget to do it, multivariate testing can help you increase CRO and create more effective, attractive pages.

Leave a Comment