Multivariate Testing 101: A Scientific Method Of Optimizing Design – Smashing Magazine

Do’s And Don’ts

I have observed hundreds of multivariate tests, and I have seen many people make the same mistakes. Here is some practical advice, direct from my experience.


  • Don’t include a lot of sections in the test.
    Every section you add effectively doubles the number of combinations to test. For example, if you’re testing a headline and image, then there are a total of four combinations (2 × 2). If you add a button to the test, there are suddenly eight combinations to test (2 × 2 × 2). The more combinations, the more traffic you’ll need to get significant results.


  • Do preview all combinations.
    In multivariate testing, variations of a section (image, headline, button, etc.) are combined to create page variations. One of the combinations might be odd-looking or, worse, illogical or incompatible. For example, one combination might put together a headline that says “$15 off” and a button that says “Free subscription.” Those two messages are incompatible. Detect and remove incompatibilities at the preview stage.
  • Do decide which sections are most worthy of inclusion in the test.
    In a multivariate test, not all sections will have an equal impact on the conversion rate. For example, if you include a headline, a call-to-action button and a footer, you might come to realize that footer variations have little impact, and that headline and call-to-action variations produce winning combinations. You get a powerful section-specific report. Below is a sample report from Visual Website Optimizer. Notice how the button has more impact (91%) than the headline (65%):

    Mvt-small in Multivariate Testing 101: A Scientific Method Of Optimizing Design

  • Do estimate the traffic needed for significant results.
    Before testing, get a clear idea of how much traffic you’ll need in order to get statistically significant results. I’ve seen people add tens of sections to a page that gets just 100 visitors per day. Significant results from such a test would take months to accumulate. I suggest using a calculator, such as this A/B split and multivariate testing duration calculator, to estimate how much traffic your test will require. If it’s more than what’s acceptable, reduce some sections.

View/comment on the original post at blog @cankoklu


About cankoklu

i try to be ironic..
This entry was posted in Uncategorized. Bookmark the permalink.