Experiment Lifecycle

An experiment may progress through different lifecycle phases in Winning Variant, each designed to handle variant assignments appropriately for subjects requesting one. The progression of an experiment is typically linear. That is, once it achieves a status, it typically does not go backwards and has a clear next status.

​

Experiment Status

Like any other resource within Winning Variant, an experiment has a status that denotes its current lifecycle stage.

​

Draft

An optional status, an experiment in “draft” simply means it is not ready to go live. This allows the entity to be created, but indicates that it is either not fully configured or is not ready to be made live (such as pending implementation).

​

Active

When an experiment is marked as “active”, it is recognized by the Variant API and has variants assigned and returned for subjects as requested. An experiment actively being evaluated should be in this state until it has been proven successful in some fashion.

When a new assignment is requested, a variant will be randomly selected for the subject based on the split defined for that variant within the current cohort. For example, if a split is set to 0.500, 50% of subjects will be assigned the variant.

​

Winner Declared

The “Winner Declared” status indicates that a “winning variant” has been chosen for the experiment. This could be due to statistical significance, preference in a tie, or any other reason. This status signals that all subjects should see the winning variant.

When an experiment enters this status, all future assignment requests return the winning variant, regardless of past assignments. For example, if a subject was previously assigned the control variant, but treatment-a was declared the winner, the subject would be returned treatment-a on subsequent requests after the status change.

This status should be used for experiments that are over, but are still depended upon by technical implementations. Once there are no dependencies on these experiment being active, it should be changed to “Ended”.

​

Ended

An experiment is “ended” when assignments no longer need to be served for it, but you still may want to reference reporting for it. It’s a sort of in-between status where it’s not used anywhere, but it’s also not archived, so it’s mostly for convenience in filtering/reporting.

​

Archived

Archive an experiment when it’s long gone, after reporting is no longer needed for it. Assignment data is never deleted, so it can be referenced in the future, but archiving old experiments helps keep your labs clean.

​

Variant Assignments

Variant assignments will be managed for any experiment in the active or winner declared status.

In active, they will be assigned according to the split defined for that variant within the active cohort. If a subject is seen more than once, it will always receive the same variant it was assigned the first time.

If in the winner declared status, all subjects will receive the winning variant, regardless of the original assignment.