> ## Documentation Index
> Fetch the complete documentation index at: https://docs.winningvariant.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Experiments
> Any test you would like to run.
An Experiment is the core abstraction used to assign variants to subjects in order to *experiment* (or test) a feature, model, idea, etc. An Experiment lives within a [Lab](/api/labs) and can have its lifecycle managed independently from any other active Experiments.
## Subjects
An experiment must use exactly one [Subject Type](/resources/subjects). That is, you must define what type of identifier is used to assign and persist a variant across the life of the experiment.
For example, if you have and use a `customer_id` subject type (one which identifies unique customers in some internal database), an experiment variant will be assigned to each unique `customer_id` encountered. When that `customer_id` is seen again in the future, it will receive the same variant it did previously for the experiment. (that is, until a winner is declared - [read about the experiment lifecycle](/experiment-lifecycle)).
## YAML
### Metadata
When defining a resource in YAML, each must have a `metadata` section. The following properties within `metadata` are shared across all resource types:
| Key | Data Type | Description |
| ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | `string` | Unique identifier. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `home-page-test`. IDs will be stored and displayed in UPPERCASE. |
| `name` | `string` | Name of the resource. |
| `description` | `string` | Description of the resource. |
| `resourceVersion` | `int` | Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update. |
| `status` | [`ResourceStatus`](#resourcestatus) | The status of the resource. |
| `parentKind` | `lab` | The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments. |
| `parentId` | `string` | The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.
If an experiment does not specified a `parentID`, it defaults to `DEFAULT`. |
#### `ResourceStatus`
Possible statuses include:
| Status | Description | Resources |
| ----------------- | ---------------------------------------------------------------- | ---------- |
| `draft` | The experiment exists, but is not ready to go live. | Experiment |
| `active` | The experiment is live and making/tracking/returned assignments. | All |
| `winner_declared` | A winner has been declared. | Experiment |
| `ended` | The experiment has ended. No assignments are made/returned. | Experiment |
| `archived` | The experiment has ended and is archived from view. | All |
### Spec
Experiments exhibit the following properties:
| Key | Data Type | Description |
| ---------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `subjectType` | `string` | ID of the [Subject](/resources/subjects) to use for this experiment. If not specified, will default to `ANY`, which allows any non-empty subject ID. |
| `hypothesis` | `string` | An open field for stating the experiment hypothesis. |
| `links` | `map` | Map of links to store as metadata for the experiment. Such as links to PRDs, analysis, etc. |
| `variants` | `Variant[]` | List of variants. See below. |
| `cohorts` | `Cohort[]` | List of cohorts. See below. |
| `winningVariant` | `string` | ID of the winning variant, once status is changed to winner\_declared. |
| `endedReason` | [`EndedReason`](#endedreason) | Why the experiment was ended. See below. There’s not practical use for this field, but rather is for record keeping. |
#### `EndedReason`
| Key | Description |
| ------------------ | --------------------------------------------------------------------------- |
| `success` | The experiment was a success, achieving statistical significance. |
| `tech_issue` | There was a technical or implementation issue and the experiment was ended. |
| `no_longer_needed` | The experiment is no longer needed or relevant. |
| `no_stat_sig` | The experiment ran fine, but did not reach statistical significance. |
| `other` | |
#### `Variant`
| Key | Data Type | Description |
| ------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `id` | `string` | Unique ID for the variant. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `treatment-a`. Variants will be stored and displayed in UPPERCASE. |
| `isControl` | `bool` | Whether the variant is the control. Only one Variant may have this set per experiment. |
| `name` | `string` | The name of the variant. |
| `description` | `string` | Description of the variant. |
#### `Cohort`
| Key | Data Type | Description |
| ----------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `index` | `int` | A unique integer auto-assigned to each cohort. This field is required and must be set to the next integer in sequence for her cohorts.
If the previous cohort was `4`, the new cohort must have value `5`. |
| `variants` | [`CohortVariant[]`](#cohortvariant) | List of variants in this cohort with associated context. See below. |
| `createdAt` | `timestamp` | When the cohort was created. This will be set automatically for new cohorts. Be sure to maintain the original values for existing/past cohorts when updating an experiment. |
#### `CohortVariant`
| Key | Data Type | Description |
| --------- | --------- | ------------------------------------------------------------------------------------------------------- |
| `variant` | `string` | ID of the Variant that’s active within the cohort. |
| `split` | `float` | What percentage of traffic should see this variant. All split values within a Cohort must equal 1.000 . |
## Experiment Lifecycle
An experiment may progress through different lifecycle phases in Winning Variant, each designed to handle variant assignments appropriately for subjects requesting one. The progression of an experiment is typically linear. That is, once it achieves a status, it typically does not go backwards and has a clear next status.
Like any other resource within Winning Variant, an experiment has a status that denotes its current lifecycle stage.
### Draft
An optional status, an experiment in “draft” simply means it is not ready to go live. This allows the entity to be created, but indicates that it is either not fully configured or is not ready to be made live (such as pending implementation).
### Active
When an experiment is marked as “active”, it is recognized by the [Variant API](/api/variants) and has variants assigned and returned for subjects as requested. An experiment actively being evaluated should be in this state until it has been proven successful in some fashion.
When a new assignment is requested, a variant will be randomly selected for the subject based on the split defined for that variant within the current cohort. For example, if a split is set to `0.500`, 50% of subjects will be assigned the variant.
### Winner Declared
The “Winner Declared” status indicates that a “winning variant” has been chosen for the experiment. This could be due to statistical significance, preference in a tie, or any other reason. This status signals that *all* subjects should see the winning variant.
When an experiment enters this status, all future assignment requests return the winning variant, regardless of past assignments. For example, if a subject was previously assigned the `control` variant, but `treatment-a` was declared the winner, the subject would be returned `treatment-a` on subsequent requests after the status change.
This status should be used for experiments that are over, but are still depended upon by technical implementations. Once there are no dependencies on these experiment being active, it should be changed to “Ended”.
### Ended
An experiment is “ended” when assignments no longer need to be served for it, but you still may want to reference reporting for it. It’s a sort of in-between status where it’s not used anywhere, but it’s also not archived, so it’s mostly for convenience in filtering/reporting.
### Archived
Archive an experiment when it’s long gone, after reporting is no longer needed for it. Assignment data is never deleted, so it can be referenced in the future, but archiving old experiments helps keep your labs clean.
## Variant Assignments
Variant assignments will be managed for any experiment in the `active` or `winner declared` status.
In `active`, they will be assigned according to the `split` defined for that variant within the active cohort. If a subject is seen more than once, it will always receive the same variant it was assigned the first time.
If in the `winner declared` status, all subjects will receive the winning variant, regardless of the original assignment.
## Examples
### Create draft experiment with one cohort
```yaml theme={null}
schemaVersion: 1
kind: experiment
metadata:
id: hero-nov-2024
status: draft
name: Hero text test for November 2024
description: Test variations of the hero on the website home page. Test launches in November 2024!
parentKind: lab
parentId: marketing
spec:
subjectType: anonymous_id
hypothesis: "One of the new variants will perform better than the current static control."
winningVariant:
variants:
- id: control
isControl: true
name: Control
description: "Original, static title."
- id: treatment-a
isControl: false
name: Limited Use case
description: "Describes a limited use case to test specificity."
cohorts:
- index: 1
variants:
- variant: control
split: 0.5000
- variant: treatment-a
split: 0.5000
```
### Add a cohort to the previous example
**Cohort Index**
Note the new cohort has an `index` of `2` since it's the next in the sequence.
```yaml theme={null}
schemaVersion: 1
kind: experiment
metadata:
id: hero-nov-2024
status: draft
name: Hero text test for November 2024
description: Test variations of the hero on the website home page. Test launches in November 2024!
parentKind: lab
parentId: marketing
spec:
subjectType: anonymous_id
hypothesis: "One of the new variants will perform better than the current static control."
winningVariant:
variants:
- id: control
isControl: true
name: Control
description: "Original, static title."
- id: treatment-a
isControl: false
name: Limited Use case
description: "Describes a limited use case to test specificity."
cohorts:
- index: 1
createdAt: 2020-08-22T04:30:08Z
variants:
- variant: control
split: 0.5000
- variant: treatment-a
split: 0.5000
- index: 2
variants:
- variant: control
split: 0.7500
- variant: treatment-a
split: 0.2500
```
### Active experiment with multiple cohorts
```yaml theme={null}
schemaVersion: 1
kind: experiment
metadata:
id: hero-nov-2024
resourceVersion: 1 # Ignored on update
status: active
name: Hero text test for November 2024
description: Test variations of the hero on the website home page. Test launches in November 2024!
parentKind: lab
parentId: marketing
spec:
subjectType: anonymous_id
hypothesis: "One of the new variants will perform better than the current static control."
winningVariant:
variants:
- id: control
isControl: true
name: Control
description: "Original, static title."
- id: treatment-a
isControl: false
name: Limited Use case
description: "Describes a limited use case to test specificity."
- id: treatment-b
isControl: false
name: Expanded Use case
description: "Describes a lengthier use case to test generality."
cohorts:
- index: 1
createdAt: 2020-08-22T04:30:08Z
variants:
- variant: control
split: 0.5000
- variant: treatment-a
split: 0.5000
- index: 2
createdAt: 2020-09-22T04:30:08Z
variants:
- variant: control
split: 0.3333
- variant: treatment-a
split: 0.3333
- variant: treatment-b
split: 0.3333
```
### Experiment with winner declared
**Winning Variant**
Must be the ID of a variant defined within the experiment.
```yaml theme={null}
schemaVersion: 1
kind: experiment
metadata:
id: hero-nov-2024
resourceVersion: 1 # Ignored on update
status: winner_declared
name: Hero text test for November 2024
description: Test variations of the hero on the website home page. Test launches in November 2024!
parentKind: lab
parentId: marketing
spec:
subjectType: anonymous_id
hypothesis: "One of the new variants will perform better than the current static control."
winningVariant: treatment-a
variants:
- id: control
isControl: true
name: Control
description: "Original, static title."
- id: treatment-a
isControl: false
name: Limited Use case
description: "Describes a limited use case to test specificity."
- id: treatment-b
isControl: false
name: Expanded Use case
description: "Describes a lengthier use case to test generality."
cohorts:
- index: 1
createdAt: 2020-08-22T04:30:08Z
variants:
- variant: control
split: 0.5000
- variant: treatment-a
split: 0.5000
- index: 2
createdAt: 2020-09-22T04:30:08Z
variants:
- variant: control
split: 0.3333
- variant: treatment-a
split: 0.3333
- variant: treatment-b
split: 0.3333
```