Shop Component Merchant-Assigned A/B Testing Guide

Note

NOTE: This document is only applicable to select merchants and partners using 3rd party checkout solutions. For more information, contact our enterprise sales team.

Anchor to OverviewOverview

In order to measure the impact of Shop Pay, an A/B test will be performed to validate that the integration is having the intended effect on the key metric of conversion lift. This document aims to provide a framework and guidelines for best practices around A/B testing.

Anchor to Test ScopeTest Scope

The A/B test is intended to measure the change in conversion rate due to Shop Pay compared to control. The conversion rate funnel is scoped from the first opportunity that a customer can choose Shop Pay, through to a converted order.

With the events defined below, we will also be able to measure some other factors such as average order values and assignment distribution. Together, these various data points will help us understand the impact of Shop Pay, and sort out any discrepancies in results.

Anchor to Test ArchitectureTest Architecture

Exclusions and group assignment will be performed by your system, and you will need to instrument the required events to share select relevant data with Shopify. Below defines the key concepts that will help us ensure a successful A/B test.

Anchor to A/A TestA/A Test

Before we initiate the A/B test, we must first validate the implementation, including exclusion and assignment logic. To do this, we will perform what looks like an A/B test, but treatment will result in no change from control.

The main objectives of the A/A test are to make sure that assignment is evenly distributed between test and control and also to make sure that both groups are converting at the same rate. If we see uneven assignment distribution or if we see different conversion rates for the groups we will know there is a bias happening during assignment. Once both teams are confident that the results are accurate we can transition from A/A to A/B testing.

We should budget at least 1 week for the A/A test to confirm results and work through any discrepancies that arise.

Anchor to Javascript InclusionJavascript Inclusion

In order to perform the testing, Shopify’s JavaScript will need to be included for the duration of the user journey. This means that while our script is typically only needed where the button and email recognition components live, during testing it may need to be included in the “Thank You” / order confirmation page as well.

If you're using an existing test suite, and would prefer to use it instead of Shopify's javascript for measurement, please discuss this with your Shopify onboarding team.

Anchor to Subject IdentificationSubject Identification

For the scope of these tests, each checkout session should correspond to a unique buyer subject ID, and UUID’s representing browsers are recommended as the ideal subject identifiers.

The UUID should be unique to a buyer’s browser on a given device, and the buyer’s UUID should and can span across multiple storefront sessions. If the buyer were to visit your store with the same browser multiple times, or reloads the page, the same subject ID should be used for those sessions as well.

If your team has any concerns about subject identification, please discuss it with your Shopify contact.

Anchor to Eligibility and ExclusionEligibility and Exclusion

It is imperative that your system begins by determining any ineligible checkouts, and excluding them from group assignment. To ensure consistency of data between the merchant system and Shopify, events for ineligible checkouts must be included in the events sent to Shopify, and specified as "excluded" for the assignment_group.

While Shop Pay supports many currencies, for the purpose of A/A and A/B testing, checkouts in any currency other than the currency(ies) scoped to the test must be excluded from assignment, and Shop Pay should not be surfaced as an option for checkouts in other currencies. The merchant’s system must include events for these excluded checkouts as well, specifying them as "excluded" for the assignment_group.

Aside from currency excluded checkouts, your system should exclude any traffic that would have a bias in assignment. This should include traffic from forcing the Shop Pay on, but may also include test script/bot traffic from your developers, special customer groups like Business to Business or wholesale if they follow a different checkout path, etc.

Subject exclusion should only be used where necessary so as to avoid the exclusion process from creating bias in the results. Please discuss any exclusion logic you have in place with Shopify as soon as possible.

Anchor to Test CohortsTest Cohorts

Control:

The control cohort should be the current experience before Shop Pay is implemented. This control cohort measures the baseline conversion of the merchant shop without Shop Pay and is used to level set the results from the second cohort.

Treatment: During the A/A test, the treatment will have no effect other than firing the relevant events, the UX will be the same as the control.

During the A/B test, the treatment will expose users to the Shop Pay components, and the checkout must include both the Shop Pay button and the email recognition whenever possible.

Please note that when performing the AB test, the treatment group should only introduce the single change of adding Shop Pay. Any additional UX or feature changes added to the treatment group will invalidate the test results.

Anchor to EventsEvents

In order to facilitate this test, you are required to implement up to three events during the testing period, each firing a maximum of once per subject:

Assignment (required)
- Assignment must be performed, and this assignment event must be triggered before the checkout payment options are presented to a customer. The assignment group specified in this event for a given subject_id must be accurate, remain unchanged once set, and determine if Shop Pay will be included as a payment option for the customer during the testing period.
Checkout Begin
- This event should be fired as soon as the subject enters the checkout. It is assumed that the currency of the checkout is known.
Checkout Complete
- This event should only execute upon completion of a checkout, and should be fired immediately before or after the customer is redirected to the merchant’s confirmation / thank you page, minimizing the chance for the customer closing the window before the event is recorded.

Anchor to AssignmentAssignment

It’s important to ensure that assignment is done in a non-biased manner.

Other than for select reasons mentioned above in the Eligibility and exclusion section above, customers should be randomly assigned and never specified to be in treatment or control.

Please discuss any exclusions or other special cases with the Shopify team to ensure alignment before the testing begins.

Anchor to Payload - AssignmentPayload - Assignment

Include the following parameters in the assignment event:

subjectId: string (required)
- The unique identifier for the subject, generally a UUID representing the browser session
assignmentGroup: string (required)
- Whether the customer was assigned to the control or treatment group, or alternately excluded from the experiment.
- Valid values:
  - "control" - Shop Pay disabled
  - "treatment" - Shop Pay enabled
  - "excluded" - Not qualified, excluded from assignment
shopId: integer (required)
- A unique identifier for the merchant’s Shopify store, provided by Shopify.
experimentId: string (required)
- A unique identifier for the test, provided by Shopify.
- This should be updated after the A/A test has concluded and before the A/B test has begun
timestamp: string (required)
- Current timestamp from the client, in ISO 8601 format. (ex. "2023-11-26T07:58:30.996+0200")

Anchor to Payload - CheckoutPayload - Checkout

Include the following parameters in the assignment event:

subjectId: string (required)
- The unique identifier for the subject, generally the browser session ID. The subject_id must be assigned to an assignment_group through an assignment event before being included in any checkout events.
shopId: integer (required)
- A unique identifier for the merchant’s Shopify store, provided by Shopify.
shopPayToken: string (optional, default: null)
- Required if the checkout is completed using Shop Pay.
action: string (required)
- Whether the event was initiated from checkout-begin or checkout-complete
- Valid values:
  - “checkout-begin” - on the first page that the Shop Pay button appears
  - “checkout-complete” - on the thank-you / order confirmation page
totalPrice: string (optional, default: null)
- The total amount of the transaction processed, in the currency of the transaction and inclusive of the standard number of digits after decimal places for any given currency. (ex. USD$10.00 should be "1000")
currencyCode: string (optional, default: null)
- The ISO 4217, legacy, or non-standard code representing the supported currency of the transaction. (ex. "EUR")
timestamp: string (required)
- Current timestamp from the client, in ISO 8601 format. (ex. "2023-11-26T07:58:30.996+0200")