A/B tests (sometimes called split tests) compare two versions of a website to determine which version is more suitable for achieving a defined goal. Such a goal can be, for example, increasing the conversion rate or improving other performance indicators (KPIs). An A / B test compares two website versions by randomly directing visitors to one of the two variants and recording user behavior. This allows webmasters and companies to test how much visitor behavior is affected by changing a particular variable.
With targeted tests, you put an end to the guesswork in optimizing your website and instead make data-based decisions. “We believe …” is a thing of the past, in the future, it will be “We know …”. Measure the impact that changes have on your metrics, such as registrations, downloads, purchases, or your custom goals, to ensure that every change delivers positive results.
Quantitative data speaks for itself. So far, you and your employees can only guess how visitors to the website react to certain design elements. However, A / B testing offers you the option of showing visitors two versions of the same page. You can then use the results to determine which variant works better. Continuous testing and optimization of your website can increase sales, donations, leads, registrations, downloads, and the number of user-generated content and at the same time provide teams with valuable insights into visitor behavior.
The Origins of AB testing
A / B testing is basically a method of comparing two versions of a thing to determine which is more efficient. This process, therefore, existed long before the Internet.
Ronald Fisher, a British biologist, and statistician brought up this idea using math in the 1920s. This process made it possible to scientifically examine two different experiences. Fisher’s work advanced science enormously. A few years later, medicine began to apply the AB test principle to clinical studies.
It wasn’t until 1960 that the concept was taken up in marketing. AB testing as we know it today has only existed since the 1990s. It soon became a method appreciated by direct marketing experts. In this, a sample of consumers is presented with several versions of a message, which differ in a single criterion. Then it is measured which version has achieved better results.
Digital development has opened up new perspectives: it has brought new opportunities for testing and measuring performance. For a website, this means that A / B testing can test an almost unlimited number of versions of a page. Each version’s performance can be accurately measured using indicators such as user actions or buying behavior. Thanks to technological development, there are now also special A / B testing tools for easier execution and evaluation of such tests.
What is the Goal of A/B Tests
The aim of A / B tests is to find the best possible website version to achieve a specific goal and to learn which changes can be used to move more visitors to a website to the desired action.
Compared to the cost of paid traffic, the cost of increasing the conversion rate using A / B testing is minimal. The return on investment from A / B tests can be very high since even small changes to a website can lead to significant increases in the leads and sales generated.
Which Pages Can You Test on with A/B Testing Methodologies?
All websites can benefit from the AB test because all pages have at least one measurable goal. Whether you have an online shop, a news website, or a lead generation page – your goal is to increase your conversion rate, no matter what type of conversion it is.
Lead Generation Pages
“Leads” means qualified prospects, that is, potential new customers. Here tests can be applied, for example, to emails that are sent with the aim of increasing sales. In these cases, the A / B test includes information about the people contacted, for example, their gender or age group. AB Testing ideas for lead generation can be found here.
You can read our “Lead Generation Marketing” article to learn more.
Media, News, and Editorial Pages
With media sites, one can speak of editorial A / B tests. For example, you test the success of a content category and make sure that the content reaches the right target groups. In this area close to the press, the focus is not directly on the sales figures, but first of all on editorial success. Suitable AB Test ideas for media sites can be found here.
E-commerce and Sales Pages
A / B tests of e-commerce sites naturally deal with the sales efficiency of a site or app on which you can shop. An A / B test can be used to check which version is performing best based on the sales made. For example, the start page, individual elements of the product pages, or the ordering process (buttons, call-to-action, etc.) can be analyzed. View AB Testing ideas for e-commerce here.
How is an AB test carried out?
It is important to follow a strict testing methodology in order to achieve results with an AB test. Here are the steps to take your AB test:
1. Form a project team
The success of your tests depends not only on the AB Test Tool but also on the experience of those who are responsible for conversion optimization. Assemble your team according to the following skills: good knowledge of data analysis, skill in identifying conversion problems, and the ability to put yourself in the position of the end-user. Here you will find a checklist that you can use to put your conversion team through its paces.
Two profiles are also useful: those of the project manager and the sponsor. The project manager coordinates the teams and takes care of the road map of the tests. The sponsor supports the optimization initiatives and is responsible for the ROI. If these resources are not available, at least one contact person should be appointed to carry out the tests and evaluate the results.
2. Order the tests by priority
If conversion problems have been identified using various methods and various test hypotheses have been formulated, these must now be sorted according to priority. For this purpose, the A / B testing program and the schedule are recorded in a roadmap. The following criteria can be useful for arranging the hypotheses to be used:
- Estimated benefits: To do this, you need to analyze the potential of your solutions. What is the estimated benefit? What are the chances that you will increase your conversion rate? By evaluating your data, you can quickly identify the pages that have a high conversion potential (high bounce rate, short connection time …)
- Traffic volume on the page: It is also important to highlight the pages with the most traffic. If your number of visitors is limited, it will be difficult to determine the impact of your AB tests. It is also better to only make one change at a time, rather than plunge into tests with too many changes.
- Easy application: In order to prioritize your tests, you can also check whether the proposed hypotheses are easy to use or not. The simpler a solution (simple graphical changes, low technical requirements …), the fewer resources are required.
After arranging the hypotheses, the roadmap has to be given a shape. For this purpose, it should be put on paper and contain a maximum of information. It should also be passed on to everyone involved and the work of the latter coordinated. The roadmap also serves to control the AB tests.
3. Run the tests
The execution of a test varies depending on the selected AB testing solution and how it works.
Some AB Test Tools are complex to implement and require the assistance of technical experts to change the source code of the pages to be tested, while tests with other tools can be performed without technical knowledge. In the latter case, the user himself changes the pages of his website with a WYSIWYG editor (What You See Is What You Get). With such tools, implementation can begin faster and the user can quickly work independently.
As far as the functionality is concerned, the creation of the tests is either carried out entirely by the company or delegated to an external service provider who, in addition to his advisory function, takes care of determining the design of the variants, develops graphic and editorial elements if necessary, and finally uses the tests one of the tools available on the market.
Which AB testing solution and which mode of operation is chosen depends on the level of development and the resources of the respective company. Every company is different, so a solution must be chosen that adapts to the needs and limits of the company. A complex tool is not useful if the user wants to work independently but ultimately depends on a service provider to use it. The other way around, a tool that is too simple could only offer limited possibilities if the requirements of the company increased.
4. Evaluate the results of the tests
To evaluate the A/B Test Results, data visualization, and timing is important. Sometimes the data to be necessary to create a conclusion can be gathered from 1 million visits without a seasonal trend. Data can have different characters according to the timing and conditions of the market. To perform an A/B test and evaluate the data, the right methodology and angle have to be chosen by the company.
5. Document the tests performed
It is imperative that you document and archive your tests in order to efficiently pass on the information to all those responsible for conversion optimization. For the documentation, you should write down the following information at the end of each test:
- the name of the test
- the period of the test
- the hypothesis tested
- a description of the variants used (screenshots as help)
- the results and conclusions of the test
- the potential financial gain for one year
6. Introduce the best versions
As soon as one of the versions definitely performs better than the original version, it should be put into operation. Depending on the company, the deadline for updates can be important. In order not to miss out on any profits, most AB Test Tools suggest showing all users (100%) the best version until the changed page is online.
Then it must be checked whether the benefit found during the test is retained in the long term. There are many external factors that can explain that optimization during the test led to better performance. For example, just before Christmas, when the urgency grows, conversion rates can of course be higher. If a test then shows that the variant of a page achieves a 10% higher performance than the original page, the additional sales outside the holidays could be lower.
7. Announce the results of the tests
It is important to communicate the conclusions obtained from the tests to the managers in particular. Some results are also relevant for activities in other departments (sales, marketing, communication, etc.) – be sure to let them know. Some AB Testing Tools can evaluate and report the financial gain from testing (the difference between the revenue from the original page and the revenue from the variants). This can be used to calculate the ROI of the testing program. The investments in the program can be easily justified.
8. Test continuously
A / B testing is a continuous optimization process. Conclusions are drawn after each test, which in turn lead to new test hypotheses for the roadmap. Incidentally, the fruits of the efforts only show up after some time: The first tests certainly do not deliver the results that were initially hoped for, because it takes time to develop an expertise.
Application examples for A / B tests
Application examples for A / B testing are the optimization of landing pages, emails, or e-commerce pages with regard to conversion rate, leads, registrations, etc. The following elements can be changed and examined, among other things:
- CTA buttons: size, shape, text, color, and placement
- CTA texts: length, content, and formatting
- Headings: content, length, font size, typography
- Texts: length and content
- Images: static or carousel, size, and placement
Some of the other Marketing Terms and Guidelines you may be interested in:
- What is the ROI?
- What is NeuroMarketing?
- What is Flywheel Marketing?
- Eye-tracking and Conversions
- Community Management and Functions
- Page Impressions for Analyzing Marketing Performance
- Conversion Rate Optimization and Funnels
Which AB test should I use?
There are different types of AB tests for different sites.
- Classic AB test: The classic AB test shows your visitors two or more variants of a page under the same URL. In this way, you can measure the success of the different variants of a particular element.
- Split test or forwarding test: A split test forwards your traffic to another or several other URLs. This can be particularly interesting when you host new pages on your server.
- Multivariate test (MVT): Finally, a multivariate test can be used to measure how multiple changed elements affect the same page. For example, you can change a banner, the color of a text, or your design. You can use the test to see which combination performed best.
An AB test is possible:
- On websites: With the AB test for websites a version A of a website can be compared with a version B. The results are then analyzed according to predetermined goals: clicks, purchases, subscriptions …
- In native mobile apps for iPhone or Android: AB testing is a bit more complicated for apps. It is not possible to display two different versions of an app once the app has been downloaded to the smartphone. For this reason, numerous tools have been developed with which you can update your app immediately. So you can easily change your design and directly analyze how the changes will affect you.
- Via server-based APIs: An API is a programming interface that enables connection to software for data exchange. With APIs, you can automatically create campaigns or variants based on saved data.
AB testing and Conversion Optimization
AB testing is a tool for tracking a conversion optimization strategy, but it must not be detached from other activities. Because an AB testing tool makes it possible to statistically test hypotheses – to identify the conversion problems sufficiently, we also have to understand the behavior of the users.
The average conversion rate of online retailers is surprisingly only 1 to 3%. Why is that? Mainly because the conversion is a complex mechanism in which numerous factors come into play: such as the quality of the generated traffic, the user experience, the quality of the offer, the popularity of the online shop, or the activities of the competition.
The goal of the online retailer should always be to find and minimize any factors that could prevent visitors from shopping on their own website. There are numerous tools available to support him in this task. One of them: AB testing, which can help to make a valid decision based on data.
AB testing is a tool for tracking a conversion optimization strategy, but it must not be detached from other activities. Because an AB testing tool makes it possible to statistically test hypotheses – to identify the conversion problems sufficiently, we also have to understand the behavior of the users.
“With A / B testing you can test a hypothesis, but you cannot explain user behavior alone!” quote = “With A / B testing you can test a hypothesis, but you cannot explain user behavior alone!”
In addition to the AB test, other methods must also be used to obtain further information about users and to identify hypotheses to be tested.
Numerous sources of information can provide valuable data:
- Web analytics data: Although this data does not provide an explanation of user behavior, it does make conversion problems clear (e.g. identification of abandoned shopping baskets or cancellation of the form filling process). You can also help prioritize the pages you are testing. Checklist: How to avoid shopping cart cancellations.
- Ergonomic testing: These analyzes provide – at low cost – information about the experience with the website from the user’s perspective.
- Usability tests: While these qualitative data are limited in the number of subjects, they can include very valuable information that quantitative methods do not provide.
- Heatmaps and session recording: These methods show how users interact with the elements on a page or between several pages.
- Customer feedback: Companies collect feedback from their customers (e.g. opinions that customers leave on the website; questions that are asked to customer service). In addition to the feedback analysis, customer surveys, live chats, etc. can also be used.
How to Find Better A/B Test Ideas and Methods?
To carry out an AB test, you first need additional information with which you can identify your conversion problems and understand user behavior. This phase is particularly important and must end with the formulation of “strong” hypotheses. The above tools and similar options will help you with this task. A strong hypothesis is the first step to successful A / B testing.
The following rules apply to a correct hypothesis:
- It has to do with a clearly identified problem, the cause of which one suspects.
- It must include a possible solution to the problem at hand.
- It must indicate the expected result, which is directly related to the KPI to be measured.
For example, if the problem identified is a high dropout rate on a registration form that appears to be too long, a hypothesis could be: “Truncating the form, removing optional fields, increases the number of registered contacts.”
Which elements on your own website should you test?
What should be tested on my website? This is a frequently asked question and has to do with the fact that companies often cannot explain their conversion rate – whether it is good or bad. If a company knew that the visitors to its site did not understand the product offered, it would not primarily test the color or the placement of the button to add it to the shopping cart. Rather, it would test different formulations on customer benefits. So every case is different. We don’t want to give you a complete list of items to test here. Rather, this listing serves as a framework to identify the elements that you should test. We also give you some interesting food for thought here.
| Titles and headings Start e.g. B. by changing the headings or content of your articles to check what appeals to users more. As for the shape, changing the color or font could make a difference. | Call to action The CTA button is very important. The color, the font, the placement, and the words used (“buy”, “add to cart”, “order” …) can have a decisive impact on your conversion rate. |
| Buttons Sometimes other buttons also play a crucial role. You can make changes to the size (small, large …), shape (angular, round …) and color (black, white, colored) to attract more visitors. | Images Pictures are as important as texts. Therefore, it is advisable to try out different pictures. For example, if you run a ready-to-wear online shop, check which photos are more successful: pure product images or images on which your items are worn by models. Also vary in the size and aesthetics of your photos (hue, saturation, brightness …) and in the placement (right, left, up, down). |
| Page structure The structure of your pages must be well thought out – whether it’s your homepage or your different categories. You can add a carousel, select fixed images, change your banner, show popular products on the home page, etc. | Algorithms Use various algorithms to turn your visitors into buyers or increase their shopping cart value: similar items, most searched products … This way you can suggest items to your potential customers that are likely to interest you. |
| Business model Rethink your plan of action to get higher profits. For example, if you are selling very special goods, be sure to include other suitable products or additional services in the offer. | Forms Your forms must be clear. Test how a change in the wording of the fields, the removal of optional fields, different placement of the fields, or an arrangement in rows or columns has an effect. |
| Prices It is more difficult to do an AB test with prices. You cannot sell the same product or service at different prices. So you need some imagination to test the impact on your conversion rate here. If you offer services, you can create a savings offer with fewer options. Sell products, for example, offer a different color, shape or a different fabric. | Navigation You can test different page sequences and offer several conversion funnels that consist of one or more parts. For example, you can use one or two pages for the payment method and delivery information. |
Advantages and disadvantages of A / B tests
A / B testing is very good for testing new ideas and determining whether they lead to an improvement in the conversion rate, a reduction in the bounce rate, or an increase in the length of stay. Possible changes can be checked step by step and tested on one page. In this way, it can be checked in a comprehensible manner whether different colors, buttons, layouts, or images influence the behavior of the visitors.
A disadvantage of A / B tests, however, is the high amount of time it takes to prepare and set up two different versions of a website.
For websites with little traffic, the tests may also have to be carried out over several weeks or months in order to obtain a sufficiently large database for meaningful results.
A / B testing also cannot measure or indicate whether a website has usability problems that may be responsible for a low conversion rate.
If multiple variables are changed at the same time, there is also a risk that the results of the test will be misinterpreted.
A/B Test Tools
A whole range of professional, partly free and partly fee-based tools for A / B testing is available on the Internet:
Google Analytics Experiments
The “experiments” from Google Analytics make the tool a complete A/B test platform. Users can run split tests for up to 10 full versions of a single page.
Note: After Optimizely created, Google Analytics Experiments has been shut down.
KISSmetrics
KISSmetrics is a powerful analysis platform. The KISSmetrics JavaScript library offers a function that supports users in setting up their A/B test. Users can also integrate KISSmetrics into their internal test code or into another A/B test platform.
Unbounce
With Unbounce, responsive landing pages can be created, published and tested without any knowledge of HTML. Unbounce can be integrated into a variety of other tools. The tool also provides the ability to assign roles to team members, capture leads, and include videos, social feeds, and widgets to optimize the conversion rate of a website or email.
Optimizely
Optimizely is one of the most popular WYSIWYG(What You Sell is What You Get) A/B testing tools. Optimizely enables tests based on advertising campaigns, geography, or cookies to facilitate the design and optimization of personalized content based on target group segments.
AB testing tips and best practices
Here are some tips that can save you trouble and inconvenience. They are based on both good and bad experiences that our customers have had during testing.
1. Ensure the reliability of the data for the testing solution.
Run at least one A / A test to ensure that your visitors are randomly distributed across the different versions. This is also a good opportunity to compare the indicators of the A / B test tool with those of your web analytics tool. Pay attention to the order of magnitude here and do not worry so much about the exact correspondence of the numbers.
2. Perform a test test before starting.
Some results seem counterintuitive? Are the test settings all correct and have the objectives been correctly defined? At best, the time you spend reviewing the test will save you the time that you would otherwise have spent interpreting incorrect results.
3. Test only one variable.
In this way, the effect of this individual variable can be analyzed. If, for example, the placement of a call-to-action button and the text in the button are changed at the same time, it is impossible to determine which of the two changes led to the result.
4. Run only one test at a time.
For the same reason, only one test should be done at a time. Two tests running at the same time make it difficult to evaluate the results, especially if they affect the same page.
5. Adjust the number of variants to the traffic volume.
If there are many variants and little traffic, the test will take a very long time before there are valid results. The less traffic is assigned to the trial versions, the fewer versions there should be.
6. Wait for a 95% confidence rate before trading.
No decisions should be made before the test reaches 95% confidence. At a lower rate, the likelihood that the differences found will be due to chance rather than changes made is too great.
7. Run a test long enough.
Even if the reliability rate is reached quickly, the sample size and the differences in user behavior in relation to the days of the week should be taken into account. A test should take at least a week, ideally two. In addition, at least 5,000 visitors and 75 conversions per version should be registered.
8. If the test takes too long, stop it.
If a test takes too long to reach a 95% reliability rate, the item tested is likely to have no effect on the measured indicator. In this case, there is no use in continuing the test. Better use the traffic for another test.
9. Measure multiple indicators.
It is advisable to measure multiple goals during the test: a primary goal to determine the variations and secondary goals for a more in-depth analysis of the results. The following indicators can be measured: the click rate, the rate of adding to the shopping cart, the conversion rate, the average shopping cart value, etc.
10. Consider promotions during a test.
External variables can falsify the results of a test. Often it is traffic generation campaigns that attract visitors with unusual behavior. It is better to avoid such side effects by postponing tests or advertising campaigns.
11. Segment the target audience of the test.
In some cases, it makes little sense to run a test with all visitors to a page. If, for example, you want to test how different formulations of customer benefits affect the registration rate of a page, it is unnecessary to test the already registered users. The target audience for this test must be new visitors.
A / B testing with regard to Search Engine Optimization
Google allows and encourages webmasters to perform A / B testing, and has indicated that performing an A / B or multivariate test does not pose an inherent risk to a website’s ranking. However, it is possible to compromise the ranking if A / B test tools are misused. Google recommends that webmasters do the following to ensure that this does not happen:
A / B tests should never be misused for cloaking. Cloaking means that search engine crawlers are deliberately directed to a different version of the website than users in order to deceive the search engine. But even if webmasters have no such intentions, they should be careful not to fundamentally change the content of a page as part of A / B tests. If Google determines that the variation of a page deviates significantly from the original, not only in terms of design but also in scope and content, Google could misunderstand this change as cloaking and the website may be subject to a penalty. For this reason, an A / B test should never last longer than necessary, since Google could also perceive this as an attempt to deceive.
Canonical tags should also be used in A / B tests to refer to the original page if the A / B test has multiple URLs.
Setting up 302 redirects is not harmful unless they redirect to unexpected or unlinked content. However, 301 redirects should not be used as they would signal Google that a permanent redirect is unusual in A / B tests.
- B2P Marketing: How it Works, Benefits, and Strategies - April 26, 2024
- SEO for Casino Websites: A SEO Case Study for the Bet and Gamble Industry - February 5, 2024
- Semantic HTML Elements and Tags - January 15, 2024
