Testing and analytics tools have made it fairly easy to run tests, but they don’t make it easy to identify what to test.

A sound testing strategy that takes the current business goals, limitations, and possibilities into account is vital for a successful testing program.

Heads up! This guide assumes you already have key performance indicators (KPIs).
These guidelines and strategies make the assumption that key performance indicators have been properly set up for measurement and that funnels are clearly defined and tracked. If that is not the case, start by learning more about defining goals and funnels and proper analytics program setup.

Identifying Underperforming Areas

Your current analytics tools can provide a picture of your overall site health and this can be a good place to start when identifying underperforming areas. However, analytics does not always tell you WHY users do not complete the purpose you want them to, such as checking out or signing up for an account. You will need to first define what your business goals are. If your goals are mainly to get users to sign up for an account, that’s very different than needing to move users through the conversion funnel. It’s very important to first establish business goals for the test and then you can identify the areas where these goals are affected. By first identifying areas of the site impacted and then taking a closer look at which aspects are underperforming is the best approach to help maximize time and effort, instead of focusing on areas of the site not impacted by the direct goal you are trying to achieve or areas that are currently performing as expected. Prioritize the pages with the highest potential for improvement that that can help you achieve your goals.

Identifying Proper Test Metrics

After clearly defining your goals, you will then need to decide the key performance indicators (KPIs) important to you for the purpose of the test. A KPI is very different from a business objective. A KPI is how you will define reaching that goal. It’s also a good plan to have everyone involved define the metrics/KPIs the same way and use agreed upon definitions, as well as to agree on a Success Metric. A Success Metric is how you will define which version of the site is successful (the winner). If your goal is to get users to become members by signing up for an account, your success metric could be the highest number of users who filled out the account form on whichever version.

Assuming KPIs are in place, the most common testing metrics are:

  • Bounce Rate: users who land on your site and leave without any other action
  • Exit Rate: users who leave the page they landed on to further explore your site
  • Engagement: also called Time on Site, this metric measures the average time users spend on your site and on various pages. This is essentially measuring how engaged they are on those pages but you must be careful with this metric because it can sometimes mean users are confused.
  • Conversion Rate: this is usually converting casual users to paying customers but can also be used to describe any action such as converting users into members signing up for an account, newsletter, etc.

Considering Constraints

There are several ways the test plan can encounter constraints and it’s important to have those expectations before getting started. Some testing constraints can include branding (How much creative leeway do you have within the branding guidelines?), resources (Is your creative or IT team available if needed? Are you in the process of changing any pages on the site that aren’t being tested already?), and traffic. For a test to be successful, you must have adequate traffic to achieve statistical confidence. If enough traffic is not exposed to the test, it will have to run longer than originally planned, or a different testing method will need to be used, or it will never achieve statistical confidence. These are all things to consider before getting started so that expectations can be managed.

Next Steps

Now that you understand better how to identify underperforming areas, nail down test/success metrics, and have considered the constraints, let’s move on to learning more about testing types and test validation methods.