WordPress A/B testing

If you spend enough time working with WordPress, you will end up developing a sixth sense for what works and what doesn’t. For example, experience enables us to distinguish good blog post ideas from terrible ones, or a great call to action from a shoddy one. Why rely solely on your gut though, when you can implement a WordPress A/B testing strategy and get some hard, decision-making data?

When done right, A/B (or split) testing takes the guesswork out of the equation, and will empower you to make the best possible decisions for the growth of your site, blog, or ecommerce store.

We’ll walk you through a couple of reliable WordPress A/B testing tools later on in this post, but first let’s take a minute to discuss the basics of split testing, and some of the best practices you should observe to maximize their efficacy. 📈

What is A/B testing and what makes for a good test?

As its name implies, an A/B test involves pitting two options (A and B) against each other, and seeing which one performs better at the end of a set period or threshold. When it comes to websites, this usually involves two versions of the same page with a key difference, such as a variation of a call to action, a different headline, or an alternate color scheme. These are all valid types of split tests.

After selecting your two versions, you must state a hypothesis and choose a goal. For example, if you tested two calls to action, you could say that whichever one accumulated the most clicks is better from a conversion standpoint. That would be your hypothesis, and obtaining the most clicks would be your goal.

Visitors would then be shown either page randomly when they browse your site, and you’d wait until your sample size was large enough to be considered statistically reliable. Once it is, you could see which call to action came out on top by comparing clicks.

It’s pretty simple 😎 – even more so if you’re using a WordPress A/B testing plugin. However, before we jump into creating some tests, let’s discuss A/B testing best practices:

  1. Make sure your variations are different enough to avoid confusion.
  2. Set a clear hypothesis and goal for your test before starting.
  3. Select your pool of testers randomly.
  4. Make sure your test runs until you accumulate a statistically significant result.

As a WordPress user, you are regularly faced with options that could be A/B tested, so internalizing the aforementioned steps is fundamental. If you formulate the right hypotheses, a well run split test will tell you exactly how you need to proceed.

WordPress A/B testing: what kind of solutions are out there?

Let’s find out  how to implement your own split tests in WordPress using two simple plugins,  each of which offers free and premium versions.

  1. Nelio AB Testing
  2. VWO

1. Nelio AB Testing

Nelio AB Testing is a comprehensive WordPress A/B testing solution that pairs a pleasant interface with an abundance of features. It enables you to run almost any type of test you want. You can split test entire pages, headlines, theme variations, widgets, and much more.

The plugin’s only downside is its view-based pricing system. The free version enables you to display your tests to up to 1,000 viewers. You can increase this limit by referring new users. Even the premium version offers view limits that are too small for sites that receive lots of traffic, thus forcing them into purchasing higher tiers.

Setup instructions ⚙️

After installing and activating the plugin, and navigating to its main screen, you’ll see four options:

The split testing options available on Nelio AB Testing, a popular plugin used for WordPress A/B Testing.

For simplicity’s sake, let’s choose New A/B Test for Posts, and select Headline Experiment – it’s Nelio’s simplest split test option. It will enable you to familiarize yourself with its setup process nicely. For starters, you’ll need to set up a name to keep track of your experiment, a description, and select which post to use as a base:

Setting your Headline Experiment details.

The following screen will enable you to create multiple variations of the same post or page with different headlines. Unleash your creativity, then click Create:

Creating new headlines on Nelio AB Testing.

Now navigate to your Experiments tab and click on Start below your test:

Starting your Nelio A/B test.

…and once complete, your results will appear on a screen like this one:

Your Nelio test results screen.

Pricing 💳

Free, $29, $89, and $259 per month.

Should you consider Nelio AB Testing? 🤔

Definitely – if you don’t mind paying for a premium service. Its free plan is incredibly limited, but will enable you to familiarize yourself with the inner workings of the plugin and help you make a decision.

2. VWO

VWO is an excellent tool for WordPress A/B testing and this is their landing page.

VWO is a free tool that focuses on helping companies find the best solutions to increase conversions and get insights about user behavior. This can be done with the advanced A/B testing feature that VWO is providing. In the free version, you can test up to 50,000 unique visitors in a month.

With VWO, you can A/B test any metrics, measure their impact, compare multiple combinations, and run many experiments simultaneously. It delivers a nice visual editor for marketing teams to create test variations easily.

The premium version of the tool lets you analyze user behaviors through session recordings, heatmaps, on-page surveys, funnels, form analytics, mobile app testing, and more options.

Setup instructions ⚙️

Sign up for the VWO plan that suits your needs. Then, install the Visual Website Optimizer plugin for free to automatically integrate VWO with your WordPress website. Copy and paste your account ID to your WordPress dashboard (Settings -> VWO) and you’re set.

To start the A/B testing, simply go to your VWO dashboard -> Testing -> A/B, and click on Create an A/B Test.

create a/b test with VWO tool

The next step is to define what you want to test (the URLs of the pages you want to compare) and the hypothesis of the experiment. After adding the pages, you will be redirected to the visual editor of the tool where you design variations of the pages you want to compare.

VWO visual editor

Next, define the metrics that you want to track. The tool can track new page visits, form submissions, engagement, or clicks.

VWO - set the A/B Test metrics

Click Next, followed by Create, then either start your campaign right away or schedule it for later. To collect insights and results, go to the same A/B section in your VWO dashboard and you will see your campaign listed there.

VWO A/B Test reports

Pricing 💳

VWO’s pricing model is set up in tranches that are divided by monthly tracked users – the more users, the more money you pay. The lowest tranche supports up to 10,000 users. On top of that, there are four different plans – starter, growth, pro, and enterprise – which all come with more advanced features as you move up the ladder. At the 10,000 user level, the prices for the four plans are as follows:

  • Starter: Free
  • Growth: $200
  • Pro: $480
  • Enterprise: $1,068

The amounts above reflect the monthly rate, but VWO bills you on annual basis. If the cost seems out of reach, it’s worth repeating that the free plan supports up to 50,000 monthly users and is a legitimate plan. It’s not one of those we-barely-give-you-anything-so-that-you-are-forced-to-pay deals. The vast majority of users will find it sufficient for their WordPress A/B testing needs.

Should you consider VWO? 🤔

VWO is a go-to solution for both growing and long-established companies because it offers complex features in the premium plans. At the same time, if you are just starting out, the free version offers plenty of ways to learn more about your business.

Conclusion 🧐

At first sight, implementing a WordPress A/B testing plugin is not as complicated as it may appear. As we’ve already established, there are plenty of tools at your disposal, and a vast array of guides to walk you through the finer points of split testing (including this one!).

👉 To make sure your test is worthwhile, follow these simple guidelines:

  1. Make sure your testing criteria is different enough to avoid confusion.
  2. Set a clear goal and hypothesis for your test.
  3. Select a random pool of testers.
  4. Give your test enough time to accumulate a decent sample size and ensure your results are statistically significant.

Are you a fan of WordPress A/B testing, and do you think it’s important for conversions? Tell us why in the comments section below!

Free guide

4 Essential Steps to Speed Up
Your WordPress Website

Follow the simple steps in our 4-part mini series
and reduce your loading times by 50-80%. 🚀

Free Access

0 Comments
Inline Feedbacks
View all comments

Or start the conversation in our Facebook group for WordPress professionals. Find answers, share tips, and get help from other WordPress experts. Join now (it’s free)!