Manage Your Experiments lets you run A/B tests (also known as split tests) on your brand’s listing content. Experiments help you compare two versions of content against each other so you can see which performs better. At the end of an experiment, you can review which version of content performed the best and then publish the winning content. By running experiments, you can learn how to build better content that appeals to your customers and helps to drive more sales.
What are good experiments?
The best experiments share these characteristics:
Version A and Version B are very different from each other. For product titles, try significantly shortening the title length to reduce noise and encourage more customers to visit your detail page. For image experiments, try alternatives that make your product easier to understand and more information-rich.
Important: The more that your content and treatments are different, the more likely it will be that any performance differences detected are meaningful and not caused by random chance.
The duration of the experiment is 10 weeks. This allows us time to collect as much data as possible.
Set up an experiment
Step 1: Start an Experiment: From the Manage Your Experiments main screen, click on the Create a New Experiment drop-down. Then select your desired experiment type.
Step 2: Select an eligible ASIN to experiment on: You will be prompted to select a Reference ASIN. A ‘Reference ASIN’ is the product that you will create experimental content for.
You may choose a variational ASIN for experimentation. If you choose a variational ASIN, the system will detail which child ASINs can be included in the experiment. The system will display ASIN eligibility status, along with details about why some ASINs may be ineligible.
Note: ASINs with low traffic may not show up in the available ASINs list and are not eligible for experimentation.
Step 3: Add Experiment Details: To create your experiment, enter the following details:
Experiment name: This name will only be visible to you. This name is important as you will use it to identify your experiment when it is running to review the results.
Hypothesis: A hypothesis is one of the most important parts of your experiment. Your hypothesis asks a question that you expect to evaluate with your experimental content. An example hypothesis could be ‘Changing my product title from being vague to being more descriptive will drive more sales’. By stating and validating the hypothesis, you can work to create learnings that you can apply to products beyond those under experimentation.
Experiment duration and start dates: The recommended experiment duration is 8-10 weeks. You can always change your duration or end your experiment early. However, the longer you run your experiment, the more confident you will be in the results.
Note: Depending on the validation time of the experimental content type, the earliest start day may be several days into the future. This gives Amazon time to validate that all submitted content meets our guidelines.
Step 4: Select experimental content: Based on the experimental content type, select content this way:
Product Titles: Enter your proposed titles for experimental content into each associated box.
Product Images: Click on Upload image and then use the file picker to select a compliant product image.
For variational ASINS, you can submit product title or product image content for some or more of the child ASINs, but you always have to submit content for the parent. This is because the parent ASIN’s content is used more broadly across the customer experience.
Typically, you will have to create new content to use as Version B. The easiest way to do this is to click on the link on our page that says ‘Start by duplicating Version A’. That will create a copy of your Version A content with the same set of ASINs (both versions of content must have the same ASINs applied to submit a valid experiment).
Step 5: Submit your experiment: At this point, your experiment will be scheduled pending content validation.
Important: Make sure to return to Manage Your Experiments in the days after you have submitted your experiment to validate that content validation passed. If it failed (e.g. submitting an image with a non-white background in a category where this is required), modify the content to be compliant with the failed validation and submit your experiment again.
Any content submitted during an experiment must meet the same guidelines as any regular content not part of an experiment.
Specific guidelines for experimental content types is as follows:
Product Titles: Product titles must not have more than 200 characters, including spaces. This upper limit applies to all categories. Some categories might have a limit of even fewer characters.
Product Images: Images are very important to customers, so quality matters. Choose images that are clear, easy to understand, information-rich and attractively presented.
Edit an experiment
To edit an experiment, start by viewing the experiment details. There are multiple states that your experiment can potentially be in, and these states will affect what detail can be edited. Your experiment can be edited in these states:
Scheduled, waiting content validation: In this state, you can edit any of your experiment details including its content.
Failed content validation: In this state, content that you submitted failed validation and you need to revise your experimental content for your experiment to run.
Scheduled, successful content validation: In this state, your content has passed validation and your experiment schedule is locked. If you choose to change the experiment contents, you must select a new experiment start date to allow time to repeat the content validation process.
Experiment in progress (partial editing available): While an experiment is in progress, the content can no longer be revised. The hypothesis and duration of the experiment can still, however, be revised as desired. If you want to change the experimental content once the experiment is in progress, cancel the existing experiment and create a new one.
Experiments in these states cannot be edited:
Cancelled: Your experiment was terminated prior to the end date.
Completed: You experiment reached the end date. At this point, all customers will see your original (control) content again until you make a decision about what content to publish.
Important: Regardless of the experiment results, when your experiment ends, your experimental content will not publish automatically. Make sure to publish your experimental content if you want customers to see it post-experiment.
Cancelling an experiment
To cancel an experiment, start by viewing the experiment details. Then click on Cancel Experiment and indicate that you are sure you wish to cancel. Once cancelled, the test will be over, and all customers will revert back to seeing the original content. The system will prompt you for the reason that the experiment was cancelled. For example, you might indicate cancellation was due to an experimental content error or that the experiment realised its desired result before its end date.