How To Create A Hypothesis Test

Create an A/B test

Follow these steps to create a simple A/B test.

An A/B test is a randomized experiment using two or more variants of the same web page (A and B). Variant A is the original and variant B through n each contain at least one element that is modified from the original.

Before you can start testing, you need something to test on. Rather than randomly testing items on your homepage or adapting your checkout flow, start small. Test a change to a Call To Action (CTA), change the color of a button, or remove an extraneous form field. Once you're comfortable creating variants and experiments, you can expand the scope of your testing.

Find more idea starters in the ideas section of the Optimize resource hub.

Create a hypothesis

Before creating your first experience you need to identify a problem, then create a hypothesis (backed up by data, of course) about what you can change to improve it.

What's the problem that you want to solve? Have conversions dropped off? Have traffic patterns changed? Have your demographics shifted? A close examination of trends in your Google Analytics behavior reports is a great place to start.

Once you've identified a problem, assemble a team within your organization and solicit their opinion about the cause of the problem. Use feedback from this cross-functional team to form your hypothesis, an educated guess that you'll validate or invalidate with experimentation.

Example hypothesis

"Changing the color of the 'Add to cart' button from blue to green will increase revenue by 10 percent."

After you've identified a problem (low conversions), and worked with your team on a hypothesis (changing the button color) you're ready to test your hypothesis on your website.

Create an A/B test

In this experiment, we'll test changing the color of a button:

Go to your Optimize Account (Main menu > Accounts).
Click on your Container name to get to the Experiments page.
Click Create experiment.
Enter an Experiment name (up to 255 characters).
Enter an Editor page URL (the web page you'd like to test).
Click A/B test.
Click Create.

The Optimize

Use a fully resolved URL in the editor page. Redirects aren't supported here.

The variants card

The top of the experiment page includes the variants card. This where you'll create the specific changes to your web page that you wish to test, called variants. You can create as many variants as you wish to test against your original page (the Editor page in Optimize).

Create a variant

To get started, click Create variant (bottom right), enter a variant name, then click Add. Repeat this process to create additional variants. When finished, you'll see a list of your new variants on the variants card.

To start making changes. click anywhere in the variant row (which will say "0 changes"). This will launch the Optimize visual editor – an overlay on top of your editor page consisting of two components: the app bar (at the top of the page) and the editor panel (floating in the lower right).

Start editing:

Click on any web page element you wish to edit (e.g. a button).
Use the editor panel to make a change (e.g. change the button color).
Click Save.
Continue making edits as necessary.
Click Done.

Variant weighting

All variants are weighted equally by default in Optimize. A visitor who is included in your experiment has an equal chance of seeing any of your variants. If you want to direct more, less, or even all of your traffic to a specific variant, you can adjust your variant weights on the experiment details page.

The configuration card

Objectives tab

Configure your experiment objectives on the Objectives tab:

Select a Google Analytics view.
Select a Primary Objective.
(Optional) Click + Add objective to add an objective. Learn more.
Add a description and hypothesis.
Click Save.

Targeting tab

Configure who and when to target on the Targeting tab.

Who to target

The Who section of the Targeting tab is used to select the visitors to whom you wish to target your variant. Enter a numeric value (in tenths of a percent) or use the slider to specify the percentage of your visitors to include in your experiment.

When to target

The When section of the Targeting tab determines where the experiment is shown. When is evaluated each time a user visits the experiment page. Use the rules to set where your experiment is seen. To create a targeting rule, click AND, then select a targeting rule type.

You must create either a URL or path targeting rule to start your experiment. The quickest way to get started is to create a URL matches rule with the same Editor page URL that you used when creating your experiment.

Optimize targeting

Targeting rules

Optimize includes the following targeting rule types, which you can read more about in the following articles:

URL targeting
Google Analytics Audiences targeting
Behavior targeting
Geo targeting
Technology targeting
JavaScript variable targeting
First-party cookie targeting
Custom JavaScript targeting
Query parameter targeting
Data layer variable targeting

Start your experience

Click Start, and when the status field says "running," your experiment is live on the web. Most updates happen within a minute.

How long should your experience run?

Keep an experiment running until at least one of these conditions has been met:

Two weeks have passed, to account for cyclical variations in web traffic during the week.
At least one variant has a 95 percent probability to beat baseline.

Experience management

Optimize provides several ways to manage your experiences, whether in draft, running, or ended status. This article explains how to search inside containers and how to access functions like edit, copy, stop, and archive.

Reports

To monitor a running experiment or see the results of a concluded experiment, click the Reporting tab at the top of the experiment detail page. The report is broken down into a series of cards that contain data about your experiment, including its status and how your variants perform against your objectives.

In addition to the reports included in Optimize, you can also see Optimize reports in Google Analytics. Sign in to Google Analytics, select the Reporting tab and select Behavior > Experiments in the report navigation. Learn more about Optimize reports.

In this video we'll show you how to create a new A/B test in Optimize in three easy steps, including how to create a variant, target an audience, and choose an objective.

Video: Create a new experiment in Optimize

Optimize targeting
Optimize reports
Variant weighting
Manage your Optimize experiences

Was this helpful?

How can we improve it?