How to Plan Your Website’s A/B Testing: Template and Case Study

According to a research paper published by Harvard Business Review, A/B testing is associated with an up to 15% increase in website visits and positively impacts conversions, product introductions, and other performance metrics.

Even the smallest changes can have the biggest impact. For example, Microsoft Bing used an A/B test to change the display of their ad headlines and added $100M (12%) revenue in a single year. It’s no wonder split testing remains the most popular approach in marketing.

In this article, you’ll get a complete A/B testing plan template for your website with all the required information based on Claspo’s experience driving sales and conversions.

Step 1: Measure Your Website’s Performance

Let’s imagine you are working with an e-commerce website. As a market or business owner, you’ll first want to understand how to measure your website’s performance and key metrics. This is crucial to estimate your current success and determine the results of your A/B testing campaign.

Defining Business Objectives

If you already have a website, you most likely understand its role in your business model. However, reminding yourself of the objectives is always useful for building the testing process. Start by asking yourself, “Why does my website exist?”. You’ll have to set several DUMB goals to get a good answer:

• Doable: realistic and attainable goals within the given resources and limitations;
• Understandable: goals should be clear and easy to understand, ensuring everyone involved knows what is expected;
• Manageable: goals should be broken down into manageable steps or components to prevent overwhelming your team and facilitate progress tracking;
• Beneficial: goals should provide value or benefits when achieved, aligning with overall business objectives and priorities.

For an e-commerce website, a DUMB goal would sound like this: “Within the next three months, increase the daily average visit duration on the website by 10% by optimizing the site’s layout and improving content quality.” 

Defining Website Goals

Your website goals will usually be similar to business objectives. Building on the DUMB goal of increasing the average daily visit duration on the e-commerce site, the specific website goals could be:

1. Improving user experience: you’d want to implement a more intuitive user navigation structure and faster loading times to make browsing more efficient.
2. Enhancing product display: you’d have to add high-quality images and videos for your products to provide a clearer and more appealing presentation.
3. Optimizing conversion pathways: you’d want to try simplifying the checkout process by reducing the required steps and adding pop-ups that prompt users to take action.

Now that you have your basic goals, you must understand how to assess their effectiveness.

Defining KPIs

KPIs are vital in tracking progress towards achieving the set goals. Your e-commerce website should use the following key performance metrics:

1. Cart abandonment rate: measures the percentage of shoppers who add items to their cart but leave the website without completing the purchase.
2. Average order value (AOV): calculates the average amount spent each time a customer places an order, indicating the overall value that your store gets from each transaction.
3. Repeat purchase rate: the percentage of customers who come back to make another purchase, showing customer satisfaction and loyalty to your brand.
4. Conversion rate: percentage of visitors who make a purchase, reflecting the effectiveness of the product presentation and checkout process.

Now that you understand which KPIs should be monitored, it’s necessary to understand the numbers needed to achieve your goals.

Defining Target Metrics

Setting clear target metrics is essential to work toward your goal. Here’s what you could use for the same e-commerce website:

1. Reducing cart abandonment: target a 25% reduction over the next three months.
2. Increasing average order value: target a 10% increase over the next quarter.
3. Increasing repeat purchase rate: target a 10% increase over the next quarter.
4. Achieving higher conversion rates: target a 5% increase within the next three months.

These specific goals will give your website a clear objective to strive for and an easy way to measure progress and success. You must understand the correlations between vanity and North Star metrics to determine the potential changes and business impact of your testing process.

Choosing the Tech Stack

Seeing the full picture of the situation on your website requires selecting the right technologies for gathering and analyzing data for your A/B test. There are some must-have features in each tool that you should consider:

• Statistical significance calculator: a tool for determining whether the differences in performance between variants A and B are statistically significant and not due to random chance;
• Traffic differentiation: the ability to evenly and accurately split traffic between variants A and B to make reliable test results;
• Real-Time Data and Reporting: access to real-time data to monitor test performance on the go, ensuring early detection of significant trends or issues;
• Integration capabilities: the tools should be easily integrated with other platforms like CRM, analytics, and others to use with existing data.

Let’s look at the necessary tools for your A/B test:

1. Web analytics: tools like GA4, Crazy Egg, or Mixpanel can help you analyze traffic according to multiple metrics like clicks, time on page, bounce rates, conversion rates, and more.
2. User behavior analysis: tools like HotJar or Plerdy can help you get heatmaps, session recordings, and user feedback polls to see how users interact with your website.
3. Visitor identification: tools like Leadfeeder, Factors.ai, or Albacross to identify and track accounts visiting your website, allowing you to see their countries, related companies, and the chance of a conversion.
4. Testing and experimentation: tools like Optimizely, VWO (Visual Website Optimizer), or Convert Experiences to conduct split testing, generate insights, and personalize experiences based on customer data.

Using at least one tool from each category would cost you at least $600 monthly, depending on the chosen software and subscription plan. This sum could easily reach $2000+ per month if you work with the most expensive tools, using all their capabilities.

Step 2: Prioritizing Testing Options

The next task is deciding which aspects of your website to test to effectively impact your metrics and achieve your goals. This requires prioritizing your testing options based on data to understand the most valuable sections. If you’ve analyzed your business objectives and the correlations between metrics, completing this step will be much easier.

High Potential for Improvement

Start by identifying the website’s areas with the biggest engagement and impact on your target metric. For instance, if the cart abandonment rate is higher in a certain area or stage, this indicates a need to change the shopping process. Testing could involve using urgency triggers or changing the design.

For example, Hacken.io managed to increase conversions by 200% after:

• Testing in the case studies section where they had the most traffic;
• Testing in the form where the smallest change could impact decisions.

Once the team maximized their results from these high-performing pages, they moved on to their website’s less impactful areas. 

This typically includes two types of pages:

• Low-performing pages to determine certain issues and maybe even patterns for improvement;
• Top exit pages labeled as “%Exit” in GA, indicating the last page seen by a user before leaving your website.

Having this information will help you understand which pages can benefit the most from changes, allowing you to improve the overall situation on the website.

Value and Cost

Prioritize tests based on the balance of value and cost. Start with changes that are likely to bring significant improvements without requiring substantial resources. An example could be optimizing the checkout process to reduce abandonment rates. This step typically has a high impact on sales conversion and can be achieved with a relatively low investment.

Importance

Focus on pages or features that are crucial to the customer journey. High-traffic pages, like the homepage or main product categories, are typically more influential. These pages have a direct impact on the visitor's first impression and can significantly affect the overall user experience.

It is recommended that you follow the RICE framework to determine the importance of your pages:

1. Reach: measures the number of people or the amount of traffic that the project or feature will impacted in a given time. It helps understand the scale and the breadth of the impact.
2. Impact: assesses the potential effect of the project or feature on the customer journey or business goals. Impact is typically rated on a scale (0.25, 0.5, 1, 2, 3) to denote the significance of its influence on the outcomes or objectives.
3. Confidence: measures the level of certainty about your estimates of reach, impact, and effort.
4. Effort: measures the amount of work required to implement the project, considering its complexity, costs, and time necessary to complete it.

Using this framework will help you prioritize testing options and determine the most important ones for your website.

High Traffic Volume

Pages with high traffic are essential for testing because they offer quicker insights due to the larger sample size. For example, if a specific category page brings in a substantial portion of the website’s traffic, optimizing this page could significantly improve user engagement and conversion rates. High-traffic pages also allow for faster A/B testing, enabling quicker changes and optimization.

Step 3: Testing Process

The next step in your A/B testing plan is to conduct the process properly. Now that you know the necessary metrics and the best areas for improvement, it is time to set up your test.

Forming a Clear Hypothesis

A clear hypothesis is essential for a successful A/B test. It should be a testable statement that explains what you believe is affecting a specific aspect of your website's performance. For an e-commerce website, if the issue is a high cart abandonment rate, a hypothesis might be: "Simplifying the checkout process will reduce cart abandonment rates." This hypothesis can then be tested by creating variations in the checkout process to see which one performs better.

Using Valid Statistical Methods

Statistical significance is crucial in A/B testing. It ensures that the results of your test aren’t just a random occasion. For instance, if changing the color of the “Buy Now” button increases sales, you need to be sure this change is statistically significant. 

Ideally, the statistical significance should be over 95%, providing confidence in the test results. This means that the results observed during the test are repetitive. You can prove statistical significance by:

• Increasing the sample size: if you applied changes on a single page, you could also try using them on several other pages.
• Using a control and variation group: you usually have a control group (the original version) and a variation group (the modified version). The performance difference between these two groups is what you’re testing for significance.

Generally, you must be confident that the results changed specifically because of your iterations, not the weather, time of the day, and phase of the moon.

Testing for Revenue

The ultimate goal of any product is to grow its revenue. What works for conversion rates may not always translate into increased revenue. For example, if an e-commerce site experiments with discount strategies, it's vital to measure not just the increase in sales volume but also the overall impact on profit margins.

If you see that the discount strategy doesn’t work, you might want to work more on prompting users to buy products or use special offers via pop-ups. They could act as the perfect CTA and help you increase revenue by prompting even the most indecisive buyer to make a purchase without a discount.

Easiest Testing Options

When it comes to what elements to test, start with areas that are most likely to yield results with minimal effort. For an e-commerce website, this would include:

• Product pages and landing pages: experiment with navigation, content, images, videos, and CTAs;
• Cart page: experiment with design, urgency triggers, features, and other options;
• Checkout page: experiment with urgency triggers, order bumps, different forms, and other options;
• CTA Buttons: experiment with the placement, wording, and size of call-to-action buttons;
• Pop-Ups: test different types of pop-ups around your website with prompts and special offers.

There are many other options to systematically identify and implement changes that lead to better performance, higher conversions, and increased revenue. They all depend on your industry’s specifics, but these four elements can be applied to all niches.

Step 4: Learn and Repeat

Continuous improvement is key to success. You must learn from all your A/B tests and note insights for further enhancements, as certain changes may only work with your niche and target audience.

Use Continuous Testing

As pointed out by Jonny Longden, “If you're not failing regularly, then you are either pretending you didn't fail, or you are not innovative enough.” Part of your tests will fail, and that’s completely normal. You should use continuous testing to learn from experiences and improve your website. Whether it's product displays, checkout processes, or marketing strategies, there's always room for refinement and optimization.

Expect Variance in Test Success

Not every test will yield positive results. It's common for a significant portion of A/B tests to show no improvement or even a decrease in performance. A good benchmark is that if around 20% of your tests lead to better conversion rates, you're on the right track. Each test, successful or not, provides valuable data and sets a new foundation for future improvements.

Apply Iterative Improvement

The key to effective optimization is an iterative approach. Each test gives you deeper insights into your customer's preferences and behaviors. This continuous loop of testing, learning, and applying helps steadily enhance the performance of your website. Over time, these improvements can lead to substantial gains in user engagement, conversion rates, and overall business success.

Example of an A/B Test

Let’s imagine a case study that focuses on the usage of a coupon code pop-up on an e-commerce website. This example will demonstrate how each step of the A/B testing process is applied to make data-driven decisions for website optimization.

Test A (without pop-ups)

• Setup: this test involves the original website layout without using the coupon code pop-up.
• Objective: to serve as a foundation for comparing the website's performance metrics without any additional prompts or offers.
• Metrics Monitored: conversion rate, average order value, and cart abandonment rate.

Test B (with pop-ups)

• Setup: this version of the website includes strategically placed pop-ups offering a coupon code prompting users to make a purchase with a special discount.
• Objective: to assess whether the addition of a coupon code pop-up increases conversion rates, average order value, and reduces cart abandonment rates.
• Metrics Monitored: conversion rate, average order value, and cart abandonment rate.
• Hypothesis: "Integrating pop-ups with coupon codes will increase sales due to customers using a special offer while it's still available."

Results and Analysis

• Data Collection: after running both versions for three months, data on key performance indicators were collected and analyzed.
• Statistical Significance: the results showed that Test B led to a 12% increase in conversion rates, a 7% increase in the average order value, and a 9% decrease in abandoned carts compared to Test A. These changes were tested for statistical significance to ensure they were not due to random occasions.
• Revenue Impact: it was observed that the introduction of pop-ups with a coupon code positively influenced revenue, indicating that the strategy not only improved engagement metrics but also had a beneficial financial impact.

Decisions Based on the Framework

1. Measure the Website’s Performance: initial metrics were established from Test A, providing a foundation for comparison.
2. Prioritizing Testing Options: based on website goals and KPIs, introducing coupon code pop-ups was identified as a high-potential change, especially on high-traffic pages.
3. Testing Process: a clear hypothesis was formed, and a variation (Test B) was created. The results were then analyzed for statistical significance and revenue impact.
4. Learn and Repeat: insights gained from this A/B test indicate that coupon code pop-ups are effective for this specific e-commerce website. However, continuous testing is essential. The website should continue experimenting with different types of pop-ups, offers, and placements to refine the strategy further and stay aligned with evolving customer preferences and market trends.

You can use this example as your basic framework for future A/B testing. It can be applied to nearly any area and niche, allowing you to point out the most important elements and evaluate the success of your campaigns.

Common Mistakes in A/B Testing

Frequent mistakes can reduce the effectiveness of your A/B testing process. Being aware of these pitfalls can help ensure more accurate, reliable, and useful results.

Too Many Variables

The fundamental principle of A/B testing is isolating the variable being tested to understand its specific impact. When you change multiple elements at once, it becomes difficult to understand which change influenced the results.

This may lead to confusion and inconclusive results. The best approach is to test one change at a time, which provides clear insights into how each element affects user behavior and conversion rates.

Not Allowing Enough Time for the Test

A/B tests requires time to collect enough data to provide you with statistically significant results. Finishing a test too early could lead to misleading conclusions. The duration of the test should be long enough to account for variations in traffic and user behavior over different days and times.

It's important to run the test until you have a substantial sample size that represents your audience accurately. This ensures that short-term fluctuations or anomalies don’t skew the results.

Ignoring the Test’s Context

The context in which A/B tests are conducted plays a crucial role in interpreting the results. External factors like seasonal trends, marketing campaigns, or changes in the market can significantly influence user behavior. 

Failing to consider these factors can lead to incorrect assumptions about the effectiveness of the changes being tested. Analyzing A/B test results within the broader context of your business environment, market trends, and any other activities that might impact user behavior is vital.

FAQ

• What is A/B testing?
  • A/B testing is a method of comparing two versions of a website or app against each other to determine which one performs better in terms of user engagement or conversion rates.
• How to do A/B testing?
  • To conduct A/B testing, create two versions of a page from your website (A and B), randomly show them to users, and then analyze which version performs better in achieving predefined goals or metrics.
• What are the most important aspects of an A/B test?
  • The most important aspects of an A/B test are having a clear hypothesis, ensuring a significant sample size, maintaining consistent test conditions, and using statistical methods to analyze results.
• What are the most common mistakes in A/B testing?
  • Common mistakes in A/B testing include testing too many variables at a time, not running the test long enough, ignoring external factors, and misinterpreting the statistical significance and test results.