While web analytics is the cornerstone of most data-driven online organizations today, multivariate and A/B testing are quickly taking a seat at the table. In this article, we'll take a look at:
•    how to get started
•    what site factors to test
•    how to design a website to facilitate testing
•    common errors to avoid

Getting Started With Testing
In essence, controlled experimentation is a set of methods that enable web marketers to try different versions of content and determine which are most effective at persuading site visitors to take a desired action. Common methods for running controlled experiments range from simple A/B testing to more complicated multivariate testing, also known as multivariable testing. In A/B testing, one or more new versions of a page or single site element compete against the original (control) version. For example, two new versions of a headline might compete against the original headline.

Multivariate testing, on the other hand, is like running many A/B tests concurrently, where there are multiple elements being tested at the same time. For example, two alternate product images, plus two alternate headlines, plus two alternate product copy text, for a total of 27 possible combinations (including the original control versions).

What's important to understand about multivariate testing is that it not only shows you which combination of elements generate more sales or pull more leads, but also reveals which individual elements influence visitor behavior versus those that do not.
 
What to Measure: How to Define Test Objectives
Before you actually start running tests for the first time, the most important step is to ensure that there are well-defined and measurable objectives for your website. For example:
1.    Generate revenue: This could mean selling product, creating leads, or referring visitors to other sites (e.g. for affiliate commissions).

2.    Cut costs: This might mean enabling visitors to use self-service features and/or answer product and service questions on their own (such as through online documentation and FAQs).

3.    Create brand awareness, which could mean thought leadership and industry visibility.

Once you have determined your website goals, the next step is deciding upon which key performance indicators (KPIs) are best. These KPIs should measure progress towards your specific marketing goals. For example:
•    If the goal is to generate revenue, you might want to track what users click on in the conversion funnel, such as the "Buy Now" button and resulting "Thank You" pages.
•    If the goal is to cut costs, you might consider tracking the interactions with both the site’s self-service areas (like FAQs and help content) as well as non-self-service areas (such as contact and help ticket generation).
•    While brand awareness goals can be more difficult to track, certain KPIs can be used as proxies for customer loyalty; for example, recency and frequency of customer visits and time spent on site, as well as the percentage of returning visitors.

Once you know your website goals and their corresponding KPIs, it’s time to decide upon the goals of your testing strategy. Your testing strategy goals are often related to your website goals, but can be much more granular.

Deciding What to Test
Similar to testing in the offline world, a large part of what can be optimized on websites has to do with marketing and promotional messages, i.e. copy writing. But beyond that, there’s a whole world of user interface and navigational elements that can be optimized as well.

Here are some examples of what you can test on your website:

Text/copy
•    Long versus short
•    Style and tone, such as chatty versus formal
•    Positioning, such as which value propositions, features, and benefits work best
•    Call-to-action text

Font, color, and size
•    Think about your target audience. For example, if your audience is older, test larger fonts

Buttons
•    Color
•    Location
•    Text/copy

Navigation
•    Sequence of items
•    Labels
•    In-line text links
•    Link style, such as underline or bold

Images
•    Content of image, such as including a man versus a woman versus a group
•    Size
•    Location on page

Layout
•    Location and size of areas or boxes on the page
•    Attention focus, such as which layouts help focus versus create distraction
•    Buttons and links, and where to place them

Functionality
•    Three-page checkout versus five-page checkout
•    Requiring form fields versus making them optional
•    Adding bells and whistles versus turning them off

Considerations in Website Design
Note that some sites can be made easier to test than others, and doing so will yield a competitive advantage that makes it possible to out-test, out-learn, and out-optimize competitors.

Here are three tips to prepare your site for multivariate testing, which represent the fundamentals of making content “testable”:

1) Use cascading style sheets (CSS)
CSS is a popular choice for web design consistency and standardization, and that's great news for multivariate testing. Among other things, CSS centralizes site-wide styles such as text and headline specifications (font, style, color, size), page layout, and positioning of elements – this makes it much easier to test multiple versions of site elements.

2) Employ text-oriented navigation
In the past, web designers often relied on images or Flash for navigation text elements. Unfortunately, using images or Flash for navigation has the side effect of making it difficult to test alternate text labels — for example, to test "My Preferences" vs. "My Profile" vs. "My Settings." Over the last few years, however, browsers have become much better at supporting most CSS standards, so there's little reason anymore to rely on images or Flash for navigation. Text links are much easier to modify and control, and thus are much easier to test and optimize.

3) Apply fluid design
Another important tip to keep in mind when creating or overhauling a site's design is to use fluid design wherever possible. Fluid design is a general approach where a site's elements can be added, removed, or rearranged without interrupting or breaking the presentation of the surrounding content -- and makes multivariate testing much easier because you can determine which layout is most effective.

Common Errors to Avoid
Once you’ve identified your website goals, KPIs to measure, the elements to test, and you have a test plan in place, you’re ready to actually launch a test.  But take note of these five common mistakes:
1)    Improper factoring: This means poor (or no) isolation of individual changes, for example, changing a headline’s text, font color, and size, all at the same time. For example, these should be 8 versions, not 2:

Improper factoring is problematic because it makes it difficult (or impossible) to isolate the impact of each individual change – was it the font color, size, or copy that cause the visitor to behave differently?

2)    Running a multivariate test too short/long – This means stopping a test early because you think you have a winner or letting a test run too long because you don’t yet have a clear winner. Unfortunately, running a test too short increases the risks of false positives and false negatives while running a test too long wastes time waiting for marginal results and consumes sample that could be applied toward another test.

3)    Tracking or analyzing wrong KPIs – This means measuring key performance indicators that are too far upstream (in the funnel) from the ultimate goal or measure only one KPI when there are several that matter. There is always the risk that the measured KPI improves at the risk of another (untracked) KPI or that the KPI being measured is actually a bad predictor of the ultimate goal.

4)    Not targeting or segmenting – When you optimize your site for everyone, you’re optimizing for no one. You should target tests to include good visitors and exclude bad visitors and segment results. That’s because not all visitors are the same; they’re in different stages of the buying cycle and some are in the wrong place altogether!

5)    Not taking action on results -- When you don’t make the winning changes to your site, or not taking what you’ve learned to run another test (that iterative “test-learn-repeat” process), you’ll gain no real momentum, and likely suffer from underwhelming ROI on your testing program.