Experiments

An Experiment is the core abstraction used to assign variants to subjects in order to experiment (or test) a feature, model, idea, etc. An Experiment lives within a Lab and can have its lifecycle managed independently from any other active Experiments.

Subjects

An experiment must use exactly one Subject Type. That is, you must define what type of identifier is used to assign and persist a variant across the life of the experiment.

For example, if you have and use a customer_id subject type (one which identifies unique customers in some internal database), an experiment variant will be assigned to each unique customer_id encountered. When that customer_id is seen again in the future, it will receive the same variant it did previously for the experiment. (that is, until a winner is declared - read about the experiment lifecycle).

YAML

Metadata

When defining a resource in YAML, each must have a metadata section. The following properties within metadata are shared across all resource types:

Key	Data Type	Description
`id`	`string`	Unique identifier. Must be lowercase and consist of alphanumeric characters, ’-’, ’_’ or ’.’ For example: `home-page-test`
`name`	`string`	Name of the resource.
`description`	`string`	Description of the resource.
`resourceVersion`	`int`	Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update.
`status`	`ResourceStatus`	The status of the resource.
`parentKind`	`lab`	The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments.
`parentId`	`string`	The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.

`ResourceStatus`

Possible statuses include:

Status	Description	Resources
`draft`	The experiment exists, but is not ready to go live.	Experiment
`active`	The experiment is live and making/tracking/returned assignments.	All
`winner_declared`	A winner has been declared.	Experiment
`ended`	The experiment has ended. No assignments are made/returned.	Experiment
`archived`	The experiment has ended and is archived from view.	All

Spec

Experiments exhibit the following properties:

Key	Data Type	Description
`subjectType`	`string`	ID of the Subject to use for this experiment.
`hypothesis`	`string`	An open field for stating the experiment hypothesis.
`links`	`map<string,string>`	Map of links to store as metadata for the experiment. Such as links to PRDs, analysis, etc.
`variants`	`Variant[]`	List of variants. See below.
`cohorts`	`Cohort[]`	List of cohorts. See below.
`winningVariant`	`string`	ID of the winning variant, once status is changed to winner_declared.
`endedReason`	`EndedReason`	Why the experiment was ended. See below. There’s not practical use for this field, but rather is for record keeping.

`EndedReason`

Key	Description
`success`	The experiment was a success, achieving statistical significance.
`tech_issue`	There was a technical or implementation issue and the experiment was ended.
`no_longer_needed`	The experiment is no longer needed or relevant.
`no_stat_sig`	The experiment ran fine, but did not reach statistical significance.
`other`

`Variant`

Key	Data Type	Description
`id`	`string`	Unique ID for the variant. Must be lowercase and consist of alphanumeric characters, ’-’, ’_’ or ’.’ For example: `treatment-a`
`isControl`	`bool`	Whether the variant is the control. Only one Variant may have this set per experiment.
`name`	`string`	The name of the variant.
`description`	`string`	Description of the variant.

`Cohort`

Key	Data Type	Description
`index`	`int`	A unique integer auto-assigned to each cohort. This field is required and must be set to the next integer in sequence for her cohorts. If the previous cohort was `4`, the new cohort must have value `5`.
`variants`	`CohortVariant[]`	List of variants in this cohort with associated context. See below.
`createdAt`	`timestamp`	When the cohort was created. This will be set automatically for new cohorts. Be sure to maintain the original values for existing/past cohorts when updating an experiment.

`CohortVariant`

Key	Data Type	Description
`variant`	`string`	ID of the Variant that’s active within the cohort.
`split`	`float`	What percentage of traffic should see this variant. All split values within a Cohort must equal 1.000 .

Examples

Create draft experiment with one cohort

schemaVersion: 1
kind: experiment
metadata:
  id: hero-nov-2024
  status: draft
  name: Hero text test for November 2024
  description: Test variations of the hero on the website home page. Test launches in November 2024!
  parentKind: lab
  parentId: marketing
spec:
  subjectType: anonymous_id
  hypothesis: "One of the new variants will perform better than the current static control."
  winningVariant:
  variants:
    - id: control
      isControl: true
      name: Control
      description: "Original, static title."
    - id: treatment-a
      isControl: false
      name: Limited Use case
      description: "Describes a limited use case to test specificity."
  cohorts:
    - index: 1
      variants:
        - variant: control
          split: 0.5000
        - variant: treatment-a
          split: 0.5000

Add a cohort to the previous example

Cohort Index Note the new cohort has an index of 2 since it’s the next in the sequence.

schemaVersion: 1
kind: experiment
metadata:
  id: hero-nov-2024
  status: draft
  name: Hero text test for November 2024
  description: Test variations of the hero on the website home page. Test launches in November 2024!
  parentKind: lab
  parentId: marketing
spec:
  subjectType: anonymous_id
  hypothesis: "One of the new variants will perform better than the current static control."
  winningVariant:
  variants:
    - id: control
      isControl: true
      name: Control
      description: "Original, static title."
    - id: treatment-a
      isControl: false
      name: Limited Use case
      description: "Describes a limited use case to test specificity."
  cohorts:
    - index: 1
      createdAt: 2020-08-22T04:30:08Z
      variants:
        - variant: control
          split: 0.5000
        - variant: treatment-a
          split: 0.5000
    - index: 2
      variants:
        - variant: control
          split: 0.7500
        - variant: treatment-a
          split: 0.2500

Active experiment with multiple cohorts

schemaVersion: 1
kind: experiment
metadata:
  id: hero-nov-2024
  resourceVersion: 1 # Ignored on update
  status: active
  name: Hero text test for November 2024
  description: Test variations of the hero on the website home page. Test launches in November 2024!
  parentKind: lab
  parentId: marketing
spec:
  subjectType: anonymous_id
  hypothesis: "One of the new variants will perform better than the current static control."
  winningVariant:
  variants:
    - id: control
      isControl: true
      name: Control
      description: "Original, static title."
    - id: treatment-a
      isControl: false
      name: Limited Use case
      description: "Describes a limited use case to test specificity."
    - id: treatment-b
      isControl: false
      name: Expanded Use case
      description: "Describes a lengthier use case to test generality."
  cohorts:
    - index: 1
      createdAt: 2020-08-22T04:30:08Z
      variants:
        - variant: control
          split: 0.5000
        - variant: treatment-a
          split: 0.5000
    - index: 2
      createdAt: 2020-09-22T04:30:08Z
      variants:
        - variant: control
          split: 0.3333
        - variant: treatment-a
          split: 0.3333
        - variant: treatment-b
          split: 0.3333

Experiment with winner declared

Winning Variant Must be the ID of a variant defined within the experiment.

schemaVersion: 1
kind: experiment
metadata:
  id: hero-nov-2024
  resourceVersion: 1 # Ignored on update
  status: winner_declared
  name: Hero text test for November 2024
  description: Test variations of the hero on the website home page. Test launches in November 2024!
  parentKind: lab
  parentId: marketing
spec:
  subjectType: anonymous_id
  hypothesis: "One of the new variants will perform better than the current static control."
  winningVariant: treatment-a
  variants:
    - id: control
      isControl: true
      name: Control
      description: "Original, static title."
    - id: treatment-a
      isControl: false
      name: Limited Use case
      description: "Describes a limited use case to test specificity."
    - id: treatment-b
      isControl: false
      name: Expanded Use case
      description: "Describes a lengthier use case to test generality."
  cohorts:
    - index: 1
      createdAt: 2020-08-22T04:30:08Z
      variants:
        - variant: control
          split: 0.5000
        - variant: treatment-a
          split: 0.5000
    - index: 2
      createdAt: 2020-09-22T04:30:08Z
      variants:
        - variant: control
          split: 0.3333
        - variant: treatment-a
          split: 0.3333
        - variant: treatment-b
          split: 0.3333

Get Started

Resources

Assignments

Subjects

YAML

Metadata

`ResourceStatus`

Spec

`EndedReason`

`Variant`

`Cohort`

`CohortVariant`

Examples

Create draft experiment with one cohort

Add a cohort to the previous example

Active experiment with multiple cohorts

Experiment with winner declared

Get Started

Resources

Assignments

​Subjects

​YAML

​Metadata

​ResourceStatus

​Spec

​EndedReason

​Variant

​Cohort

​CohortVariant

​Examples

​Create draft experiment with one cohort

​Add a cohort to the previous example

​Active experiment with multiple cohorts

​Experiment with winner declared

Subjects

YAML

Metadata

`ResourceStatus`

Spec

`EndedReason`

`Variant`

`Cohort`

`CohortVariant`

Examples

Create draft experiment with one cohort

Add a cohort to the previous example

Active experiment with multiple cohorts

Experiment with winner declared