# App Configuration Source: https://docs.winningvariant.com/advanced/app-configuration Managing and configuring the Winning Variant Snowflake Native App ## Roles A user must be assigned the `admin` [application role](/snowflake-app-details#application-roles) to execute any of the examples described on this page. ## Manage Compute Pool Winning Variant creates its own compute pool that it uses to run its services in an isolated environment. The `admin` application role is granted the following permissions: `OPERATE`, `MODIFY`, `USAGE`, `MONITOR` . This means that an `admin` can alter the compute pool directly. All services are designed to run on a single instance, so there's no point in increasing the number of nodes in the compute pool as each service will still use just a single instance. ## Manage Services The Winning Variant services can be managed by an admin using the following stored procedures (all within the `management` schema): | Procedure | Description | | --------------------------- | ---------------------------------------- | | `suspend_services()` | Suspends all services. | | `suspend_config_service()` | Suspends just the configuration service. | | `suspend_variant_service()` | Suspends just the variant service. | | `resume_services()` | Resumes all services. | | `resume_config_service()` | Resumes the configuration service. | | `resume_variant_service()` | Resumes the variant service. | | `get_service_status()` | Gets the status of all services. | | `get_service_endpoints()` | Shows ingress URLs for all services. | **Examples** ```SQL theme={null} USE ; -- Suspend all services CALL management.suspend_services(); -- Check status of services CALL management.get_service_status(); ``` ## Task Scheduling Winning Variant maintains a few internal tasks whose schedules can be modified to fit your company's needs. Each task's schedule may be modifed with the following stored procedure: `management.set_task_schedule(task_name VARCHAR, schedule VARCHAR)` | **Parameter** | **Description** | | :------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task_name` | The name of the task to reschedule. Possible values are listed below. | | `schedule` | A string defining the new schedule. Any string value compatible with the `SCHEDULE` parameter for [CREATE TASK](https://docs.snowflake.com/en/sql-reference/sql/create-task#optional-parameters) will work here. | Available tasks are: | **Task Name** | **Description** | **Default Schedule** | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------- | | `log_load` | Loads new logs into tables for immediate analysis. | At the top of each hour. | | `assignments_cache_snapshot` | Takes a snapshot of the block storage volume that contains the internal assignment cache. | Nightly at midnight Pacific Time. | | `variant_config_refresh` | Forces a refresh of the configuration used by the Variant API.

The configuration is updated in real time when an experiment is modified, but this is a "belt and suspenders" approach to make sure that it's correct at least hourly if something gets out of sync. | At the top of each hour. | **Example** ```sql theme={null} -- Load assignments once a day CALL management.set_task_schedule('log_load', '1 day') ``` ## Configuration Properties While certain components of the app are configurable directly via SQL (i.e., scaling a compute pool), other components may be configured indirectly. ### Properties | Property | Description | Default | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | | `PIPELINE:LOG_LOAD_DELAY_SECONDS` | How long (in seconds) to delay data loading. This is done to ensure that internal stages get caught up before loading data into a table. Snowflake doesn't offer an SLA for stage syncing, so this should be long enough to handle most cases. If it's too short, some data may not get picked up and will need to wait for the next run. | `30` | ### Set configuration property To set a property, execute the following procedure: ```sql theme={null} CALL management.set_config_property('', '') ``` **Example** ```sql theme={null} -- Reduce the assignment load delay to 10 seconds CALL management.set_config_property('PIPELINE:LOG_LOAD_DELAY_SECONDS', '10') ``` All values are stored as strings. So even a number, such as `100` should be stored as `'100'`. ### Get configuration properties To see the current configuration properties, perform a `SELECT` from `management.config_properties`. **Example** ```sql theme={null} -- See all configuration properties SELECT * FROM management.config_properties; ``` # App Details Source: https://docs.winningvariant.com/advanced/app-details Understand what's going on under the hood of the Winning Variant Snowflake Native App. ## Application roles The following application roles are created and can be assigned to your own roles to manage access and usage of various Winning Variant features: | Role | Description | Access | | ----------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `admin` | Administrator | An `admin` can do everything! | | `analyst` | Experimentation analyst | `SELECT` on all tables in the `experimentation` and `consumption` schemas as well as run stored procedures related to output data. | | `editor` | Winning variant resource editor | Inherits `analyst`, can use the Configuration UI, and can run all stored procedures for resource management. | | `scientist` | For users needing to read/write assignments. | Can access `experimentation.assignments` table and `experimentation.create_assignment` function. | | `proxy` | Reverse proxy for Variant API | Usage on Variant API service endpoint. This role is designed specifically for a reverse proxy sitting in front of the Variant API. | | `wvusage` | For users needing limited access to aggregated usage data. | Can `SELECT` from `experimentation.assignments_per_day` and `experimentation.assignments_per_experiment`. | ## Account-level Assets A few assets are created on the account automatically and used internally by the app. | Type | Name | Description | Auto-suspend | | ------------ | --------------- | ---------------------------------------------------------------- | ------------ | | Compute Pool | `_pool` | Single-node `CPU_X64_S` compute pool shared across all services. | 1 hour | | Warehouse | `_primary` | Single-node `XSMALL` warehouse shared by all services. | 30 seconds | The `admin` role is granted `MODIFY` on all assets to adjust sizing, if necessary. ### Warehouse utilization The `_primary` warehouse has two primary uses: 1. Resource management, the Configuration UI, and other internal tasks. These are infrequent and generally only consume resources when an asset is created/modified. 2. Loading of assignments. By default, this task runs every 30 minutes. 3. Other internal maintenance tasks, such as a daily assignment counter and daily block storage snapshots. ## Snowpark Container Services Winning Variant creates various services inside of the app, including: ### Configuration UI Your experimentation interface! See your application details or contact your Winning Variant team for your specific URL. ### Variant API The API used to get and make assignments (for experiments outside of Snowflake). [Read more here](/assignments/api). See your application details or contact your Winning Variant team for your specific URL. ### **Block Storage** The Variant API creates a block storage volume of 100GB with the lowest IOPS option by default. A snapshot is generated nightly, replacing the one from the previous day. If you need to preserve each nightly snapshot, a task will need to be confirmed to make a copy of the snapshot. **Caveat for Tri-Secret Secure accounts** ([read on Snowflake](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/block-storage-volume#guidelines-and-limitations)): Block storage volumes do not support [Tri-Secret Secure](https://docs.snowflake.com/en/user-guide/security-encryption-tss) and [Periodic rekeying](https://docs.snowflake.com/en/user-guide/security-encryption-manage.html#label-periodic-rekeying). This means that if your account has enabled Tri-Secret Secure or periodic rekeying, while all other Snowflake data will continue to have added security, any images stored in your Snowpark Container Services block storage volumes will not benefit from this added security. To create a block storage volume in an account with Tri-Secret Secure or periodic rekeying, you must first confirm that you understand and agree to continue without the benefit from this additional security for your block storage volumes. To confirm agreement, an account administrator (user with the ACCOUNTADMIN role) will need to set the account-level parameter [ENABLE\_TRI\_SECRET\_AND\_REKEY\_OPT\_OUT\_FOR\_SPCS\_BLOCK\_STORAGE](https://docs.snowflake.com/en/sql-reference/parameters.html#label-enable-tri-secret-and-rekey-opt-out-for-spcs-block-storage) to `TRUE`. ## Snowflake Credit Usage Our internal observations show that, while running and idle (no active use) under default configuration settings, the app consumes just \~3.05 credits per day. Actual consumption cannot be guaranteed and will vary upon use. # Assignments Source: https://docs.winningvariant.com/analytics/assignments Raw assignment FACT data. Each unique assignment for an experiment to a subject is logged and persistently used any time that subject is encountered again in the future. This view becomes the foundation for all experimentation reporting. ## Application Roles A user must be mapped to one of the following [application roles](/snowflake-app-details#application-roles) to access assignments: * `admin` * `editor` * `analyst` ## View View: `experimentation.assignments` | Field | Description | Data Type | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `id` | Unique ID for the assignment. This is a [KSUID](https://github.com/segmentio/ksuid) that is sorted automatically by time. | `string` | | `subject_id` | ID of the subject that given the assignment. Note that this lacks context of the actual subject type used. That will need to be done via a JOIN on the experiments table, if required. | `string` | | `experiment_id` | Experiment ID | `string` | | `variant_id` | ID for the variant that was assigned. | `string` | | `cohort_index` | Index for the active cohort at the time of assignment (the cohort the subject is in). | `int` | | `created_at` | Timestamp of when the assignment was made. ISO8601 format. | `string` | # Consumption Source: https://docs.winningvariant.com/analytics/consumption Aggregate views of assignment consumption. In order to monitor usage, the application creates views that give aggregate counts of assignments. ## Application Roles A user must be mapped to one of the following [application roles](/snowflake-app-details#application-roles) to access assignments: * `admin` * `editor` * `analyst` * `wvusage` ## Views ### Assignments per day View: `experimentation.assignments_per_day` | Field | Description | Data Type | | ------------------ | ---------------------------------------------- | --------- | | `date_created` | Date the assignment was created | `DATE` | | `assignment_count` | Number of assignments created in the given day | `INT` | ### Assignments per experiment View: `experimentation.assignments_per_experiment` | Field | Description | Data Type | | ------------------ | ----------------------------------------------------- | --------- | | `experiment_id` | ID of the experiment | `VARCHAR` | | `experiment_name` | Name of the experiment | `VARCHAR` | | `lab_id` | ID of the lab the experiment belongs to | `VARCHAR` | | `assignment_count` | Number of assignments created in the given experiment | `INT` | ## Examples **Get a daily account of assignments for the month of December 2024** ```SQL theme={null} USE ; SELECT * FROM experimentation.assignments_per_day WHERE date_created >= '2024-12-01' AND date_created < '2025-01-01'; ``` # Dimensions Source: https://docs.winningvariant.com/analytics/dimensions Resource views for use in reporting. Winning Variant will maintain a view for each resource type (lab, experiment, subject type) as they are created/updated within the platform. These may be used for joins against the primary [Assignments View](/analytics/assignments) to retrieve context for the assignments. It’s important to know that each view shows *only* the current version/state of a resources. ## Application Roles A user must be mapped to one of the following [application roles](/snowflake-app-details#application-roles) to access dimension views * `admin` * `editor` * `analyst` ## Views ### Labs View: `experimentation.labs` Unique key: `id` | Field | Description | Data Type | | ------------------ | ------------------------ | ------------------------------------------------------ | | `id` | Unique ID for the lab | `varchar` | | `resource_version` | Current resource version | `int` | | `status` | Current status | [`ResourceStatus`](/resources/overview#resourcestatus) | | `name` | Name of the lab | `varchar` | | `description` | Resource description | `varchar` | ### Subject Types View: `experimentation.subject_types` Unique key: `id` | Field | Description | Data Type | | ------------------ | ------------------------------------ | ------------------------------------------------------ | | `id` | Unique ID for the subject type | `varchar` | | `resource_version` | Current resource version | `int` | | `status` | Current status | [`ResourceStatus`](/resources/overview#resourcestatus) | | `name` | Name of the subject type | `varchar` | | `description` | Resource description | `varchar` | | `match_type` | The match type for the subject type. | [`MatchType`](/resources/subjects#matchtype) | ### Experiments View: `experimentation.experiments` Unique key: `id` | Field | Description | Data Type | | ------------------ | --------------------------------------------------- | ------------------------------------------------------ | | `id` | Unique ID for the experiment | `varchar` | | `resource_version` | Current resource version | `int` | | `status` | Current status | [`ResourceStatus`](/resources/overview#resourcestatus) | | `name` | Name of the experiment | `varchar` | | `description` | Resource description | `varchar` | | `parent_lab_id` | ID of the lab the experiment belongs within. | `varchar` | | `subject_type_id` | ID of the subject type this experiment uses. | `varchar` | | `hypothesis` | The experiment hypothesis. | `varchar` | | `active_cohort` | Index for the active cohort. | `int` | | `winning_variant` | If specified, the ID of the winning variant. | `varchar` | | `ended_reason` | The reason the experiment was ended, if applicable. | [`EndedReason`](/resources/experiments#endedreason) | ### Experiment Variants View: `experimentation.variants` One row for each experiment-variant combo. Unique key: `(experiment,id)` | Field | Description | Data Type | | --------------- | ---------------------------------------------------------- | --------- | | `experiment_id` | ID of the experiment this variant belongs to. | `varchar` | | `variant_id` | ID of the variant. | `varchar` | | `is_control` | Whether this variant is the control within the experiment. | `bool` | | `name` | Variant name | `varchar` | | `description` | Variant description | `varchar` | ### Experiment Cohort-Variants View: `experimentation.cohorts` One row for each experiment-cohort-variant combo. Unique key: `(experiment_id,cohort_index,variant_id)` . Note that experiment variants may be listed redundantly since they can exist within multiple experiment cohorts. As such, all `variant_id` values in this table are within the context of the experiment to which they belong. Within an experiment, the cohort with the highest `cohort_index` is the active one. | Field | Description | Data Type | | --------------- | ---------------------------------------------------------------------------------- | ------------- | | `experiment_id` | ID of the experiment this cohort-variant belongs to. | `varchar` | | `cohort_index` | Unique integer assigned to the cohort within its experiment. | `int` | | `variant_id` | ID of the variant (within its experiment) | `varchar` | | `split` | Percentage of traffic that should see the variant (within the cohort). E.g., 0.500 | `number(5,4)` | | `created_at` | Timestamp of when the cohort was created | `timestamp` | ## Examples **List all active experiments** ```SQL theme={null} USE ; SELECT * FROM experimentation.experiments WHERE status = 'active'; ``` # Linked Identifiers Source: https://docs.winningvariant.com/analytics/linked-identifiers Raw linked identifiers to be used for attribution in analytics. All Linked Identifiers provided via API or SQL are provided via the app to be used in analytic workloads or BI queries. ## Application Roles A user must be mapped to one of the following [application roles](/snowflake-app-details#application-roles) to access assignments: * `admin` * `editor` * `analyst` ## View View: `experimentation.identifiers` | Field | Description | Data Type | | :--------------- | :--------------------------------------------------------- | :-------- | | `created_at` | Timestamp of when the assignment was made. ISO8601 format. | `string` | | `subject_type_a` | The type of the first subject recorded. | `string` | | `subject_id_a` | The ID of the first subject recorded. | `string` | | `subject_type_b` | The type of the second subject recorded. | `string` | | `subject_id_b` | The ID of the first subject recorded. | `string` | # Experiment Summaries Source: https://docs.winningvariant.com/analytics/summary Built-in views for high-level experiment analysis. Winning Variant includes some built-in views for high-level experimentation analysis. These are designed to be generic and do not take any of your custom metrics into account. ## Cohort Summary This view shows the percentage of assignments per experiment cohort for all time. View: `experimentation.cohort_summary` | Field | Description | Data Type | | -------------------- | ------------------------------------------------------------------------------------- | --------- | | `experiment_id` | Experiment ID | `string` | | `cohort_index` | Index for the active cohort at the time of assignment (the cohort the subject is in). | `int` | | `variant_id` | ID for the variant that was assigned. | `string` | | `percent_per_cohort` | Percentage of assignments per cohort. | `number` | ## Assignments Over Time This view shows the number of unique assignments to a variant within an experiment cohort over time. View: `experimentation.assignments_over_time` | Field | Description | Data Type | | ----------------- | ------------------------------------------------------------------------------------- | --------- | | `experiment_id` | Experiment ID | `string` | | `cohort_index` | Index for the active cohort at the time of assignment (the cohort the subject is in). | `int` | | `assignment_date` | The date of assignment creation. | `date` | | `variant_id` | ID for the variant that was assigned. | `string` | | `subject_count` | Number of unique subjects assigned to the variant within the cohort. | `int` | # Assignments via API Source: https://docs.winningvariant.com/assignments/api Access Winning Variant programmatically from anywhere (including outside of Snowflake) to get or make experiment assignments. ## Overview Use the API described below to get or set experiment assignments from anywhere. This is especially useful for use cases that exist *outside* of Snowflake, such as in your app/platform. The hostname for the API can be retrieved by clicking the App from within Snowflake: Data Products > Apps > Winning Variant Experimentation. The hostname for "Variant API" is the one you'll use. ## Experiment Assignments `POST /experiment-assignments` Use this endpoint to get or set assignments for a list of subject IDs in a given experiment. If an assignment already exists, it is returned, otherwise a new one is created and returned. ### **Request Body** The request accepts a JSON-formatted body with the following parameters: | **Parameter** | **Type** | **Required** | **Description** | | --------------- | ---------- | ------------ | --------------------------------------------------------------------------- | | `experiment_id` | `string` | Yes | The ID of the experiment to get/set assignments for. | | `subject_ids` | `string[]` | Yes | List of subject IDs to get/set assignments for within the given experiment. | ### **Response** A JSON object with a single key, `assignments` , itself another object with key-value pairs. The key is the subject ID originally provided and the value is the variant it's assigned within the experiment. ### Examples **Get an assignment for a single subject** In this example, we request an assignment for user `user_123` in experiment `MY-EXPERIMENT`. The response indicates that the subject is assigned the `CONTROL` variant. ``` POST /experiment-assignments Content-type: application/json Accept: application/json { "experiment_id": "MY-EXPERIMENT", "subject_ids": ["user_123"] } Response: { "assignments": { "user_123": "CONTROL" } } ``` **Get assignments for multiple subjects** In this example, we request assignments for two users in the same experiment. The response includes the variant assignment for each, respectively. ``` POST /experiment-assignments Content-type: application/json Accept: application/json { "experiment_id": "MY-EXPERIMENT", "subject_ids": ["user_123","user_456"] } Response: { "assignments": { "user_123": "CONTROL", "user_456": "TREATMENT" } } ``` ## Public Access Snowflake currently does not allow unauthenticated public internet access to any service on its platform. Since any experiments running outside of Snowflake will ultimiately need access to the Variant API running inside of the application, you will need to deploy some sort of reverse proxy or forwarding service that takes public traffic and authenticates it with Snowflake. ### Authenticate using Programmatic Access Tokens (Recommended) The recommended approach to accessing Snowpark Container Services (including those deployed through the Winning Variant Native App) is to use [Programmatic Access Tokens](https://docs.snowflake.com/en/user-guide/programmatic-access-tokens) (PAT). Follow the documentation provided above from Snowflake to enable this for your account. Once a PAT is generated for a user mapped to the `proxy` application role, you can deploy your own simple reverse proxy that appends the `Authentication` header or use a forwarding service, such as one through [Cloudflare](https://www.cloudflare.com/). ### Authenticate using OAuth If you're unable to use Programmatic Access Tokens, you can use Snowflake's standard OAuth flow to authenticate traffic. Due to the necessary token exchanges and refreshes associated with this approach, Winning Variant has released its own open source reverse proxy, [icebreaker](https://github.com/winningvariant/icebreaker), built on top of nginx. This project handles the OAuth flow and refreshes for you automatically. In order to keep our promise of data control and privacy, we provide instructions below on how to set up this reverse proxy within your own cloud environment. Alternately, Winning Variant is happy to host this for you as a managed service offering. **1. Create a role** Create a role in your account that will be mapped to the app's role, which only has access to the Variant API endpoint. You can name this whatever you like, but we use `WVPROXY` in our examples: ```sql theme={null} CREATE ROLE WVPROXY; ``` **2. Create a dedicated user** Create a user that will be used by the reverse proxy to access the services running in the Winning Variant native app. * [Snowflake: Create User](https://docs.snowflake.com/en/sql-reference/sql/create-user) **3. Enable key pair authentication for this user.** * [Snowflake: Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth) **4. Assign your new user to your new role** For example: ```sql theme={null} GRANT ROLE WVPROXY TO USER ; ``` Alternately, you can assign this new role to an existing role in your account: ```sql theme={null} GRANT ROLE WVPROXY TO ROLE ; ``` **5. Map your account role to the app role** Within the Winning Variant app settings, map your role to the `PROXY` application role created by the app. This will give it usage of the service that serves the Variant API. Winning Variant Reverse Proxy Role **6. Deploy Reverse Proxy** Build the [icebreaker](https://github.com/winningvariant/icebreaker) project, which deploys an [nginx](https://nginx.org/) service in your own cloud region and manages the OAuth flow (including refreshes) for you automatically. This gives you complete control of the service and the data that passes through the reverse proxy while allowing you to deploy it in the same region as your Snowflake account. When deploying, you'll provide environment variables unique to your setup, including the Variant API ingress URL, your new user's username, and the private key used for authentication. **Snowflake Network Ingress Policy** If your organization enforces network policies, you may need to create an ingress policy to allow traffic from your reverse proxy. Read [Controlling network traffic with network policies](https://docs.snowflake.com/en/user-guide/network-policies) for details. For example, if your reverse proxy has an egress IP address of '10.0.0.0': ```SQL theme={null} CREATE NETWORK RULE winning_variant_variantapi TYPE = IPV4 VALUE_LIST = ('10.0.0.0/32'); ``` **Firewall** We recommend putting this service behind a web application firewall (WAF) that will help mitigate things such as DDOS attacks. Additionally, to reduce stress on the underlying Snowflake compute in the event of an attempted attack, you can limit access to the `/experiment-assignments` and `/healthz` endpoints only. **Health Checks** A health check is available at `/healthz`, which returns a `200` status code if the service is alive *and* able to communicate with the appropriate Snowflake database. This health check is used by Snowpark Container Services, but may also be used by you to monitor uptime. Winning Variant would also like to monitor the uptime of your reverse proxy, so please provide the hostname of the service, when live, and [add this list of IP addresses to your allowlist](https://uptime.betterstack.com/ips.txt). # Overview Source: https://docs.winningvariant.com/assignments/overview All about Winning Variant experimentation assignments. Winning Variant defines an "assignment" as the variant a [subject](resources/subjects) is assigned within an experiment. ## Example Imagine you have a series of experiments running on your website: | Experiment ID | Description | Subject Type | | ----------------- | ------------------------------------------------------------------ | -------------- | | `HOME-CTA` | Red/blue/green CTA button on home page | `ANONYMOUS_ID` | | `CART-NOTICE` | Show a notice after 30s to improve cart abandonment. | `ANONYMOUS_ID` | | `FEATURE-CALLOUT` | Show a new feature callout in-app and see if engagement increases. | `USER_ID` | Vanessa is a known user on your platform. When she accesses your website, she has two subject IDs: 1. Anonymous ID (`ANONYMOUS_ID`): `anon_0001` 2. Known User ID (`USER_ID`): `user_0001` If Vanessa enters all 3 experiments, she'll have 3 assignments: | Experiment ID | Subject ID | Assignment ID | Experiment Variant | | ----------------- | ----------- | ---------------- | ------------------ | | `HOME-CTA` | `anon_0001` | `assignment_001` | `RED` | | `CART-NOTICE` | `anon_0001` | `assignment_002` | `TREATMENT` | | `FEATURE-CALLOUT` | `user_0001` | `assignment_003` | `FEATURE-A` | ## Context Assignments are immutable (with a few administrative exceptions) and contain context to when they are created. Specifically, an assignment has timestamp of when it was created and the cohort number within the experiment they entered in. This allows you to properly attribute a subject's actions to their cohort and/or when they first experienced an experiment. ## Assignment Data [Read more about the raw assignment data](/analytics/assignments) that's provided from the application. # Assignments via SQL Source: https://docs.winningvariant.com/assignments/sql How to get or set experiment assignments with SQL. ## Overview Assignment can be set or retrieved directly in SQL. This is useful when doing analysis (querying directly from `experimentation.assignments` or if wanting to implement an experiment directly in SQL. ## Roles A user must be assigned of the following [application roles](/snowflake-app-details#application-roles) to execute any of the examples described on this page: * `admin` * `editor` * `scientist` ## Get assignments To get assignments for a given subject, perform a select against the `assignments` table: ```SQL theme={null} SELECT variant_id FROM experimentation.assignments WHERE subject_id = '' AND experiment_id = ''; ``` ## Get or make assignments Gets an assignment or makes one if it doesn't exist: ```SQL theme={null} SELECT experimentation.create_assignment('', ''); ``` Returns the new or existing variant assigned. ## Examples ### Manually get or create a single assignment This example gets an assignment for user with ID `user_0001` in experiment `home-cta`. ```SQL theme={null} SELECT experimentation.create_assignment('user_0001', 'home-cta'); ``` ### Get an assignment for each `user_id` value in a table ```SQL theme={null} SELECT user_id, experimentation.create_assignment(user_id, '') AS assignment FROM my_user_table; ``` ### Split test different Cortex models This example demonstrates how you can run a SQL query that produces an output from Cortex's `COMPLETE` method, testing different model versions for various customers. Winning Variant is used to determine which model should be used per customer ID. Later, we would tie business KPIs (i.e., revenue, churn, returns, etc) back to each variant to see which model should be deployed. We have the following declared for this example: * A 50/50 test between control (`llama3.2-1b` model) and treatment (`claude-3-5-sonnet`) * The Winning Variant experiment ID is `cortex-test` * The ID of *your* user in this example is `abc123` ```sql theme={null} -- Get an assignment from Winning Variant for the experiment WITH assignment AS ( SELECT create_assignment('abc123', 'cortex-test') AS variant ) -- If the assigned variant for this user is 'TREATMENT', -- use 'claude-3-5-sonnet', otherwise use 'llama3.2-1b' (CONTROL) SELECT CASE WHEN variant = 'TREATMENT' THEN SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet', 'Is a hot dog a sandwich?') ELSE SNOWFLAKE.CORTEX.COMPLETE('llama3.2-1b', 'Is a hot dog a sandwich?') END AS result FROM assignment; ``` # Deployment Source: https://docs.winningvariant.com/deployment Winning Variant can be deployed in a variety of ways inside of Snowflake or across other cloud platforms outside of Snowflake. Two things are consistent across all deployments: 1. Experiments may be implemented in any channel, application, or programming language 2. Results are made available in your BI tool of choice ## Snowflake The standard deployment has the Winning Variant native app installed in the same Snowflake account as a company's KPIs that experiments will be measured against. Winning Variant Architecture Snowflake ## Databricks For those who warehouse their data in Databricks, the Snowflake Native app's data will be pulled in via the Snowflake Spark Connector in Databricks and combined with KPIs to create the experiment result sets. Winning Variant Architecture Databricks ## Google Cloud / BigQuery For those who warehouse their data in BigQuery, the Snowflake Native app's data will be pulled in via a number of possible ELT options in GCP and combined with KPIs in BigQuery to create the experiment result sets. Winning Variant Architecture GCP ## Other Data Platforms For those who warehouse their data outside of Snowflake, the Snowflake Native app's data will be pulled in via a number of possible ELT options and combined with KPIs in the warehouse to create the experiment result sets. Winning Variant Architecture Other DP 1 ## Managed Deployment Don't want to host the Winning Variant app in your Snowflake account or don't have one? No problem! We offer a managed hosting option that runs your Winning Variant instance in an isolated account our team will manage for you. Winning Variant Architecture No Snowflake # Split Test AI/ML in Snowflake Source: https://docs.winningvariant.com/examples/ai-ml Run split tests on Cortex and ML models directly in Snowflake, testing performance against actual business metrics. ## Cortex ### Test different models with SQL This example demonstrates how you can run a SQL query that produces an output from Cortex's `COMPLETE` method, testing different model versions for various customers. Winning Variant is used to determine which model should be used per customer ID. Later, we would tie business KPIs (i.e., revenue, churn, returns, etc) back to each variant to see which model should be deployed. We have the following declared for this example: * A 50/50 test between control (`llama3.2-1b` model) and treatment (`claude-3-5-sonnet`) * The Winning Variant experiment ID is `cortex-test` * The ID of *your* user in this example is `abc123` ```sql theme={null} -- Get an assignment from Winning Variant for the experiment WITH assignment AS ( SELECT create_assignment('abc123', 'cortex-test') AS variant ) -- If the assigned variant for this user is 'TREATMENT', -- use 'claude-3-5-sonnet', otherwise use 'llama3.2-1b' (CONTROL) SELECT CASE WHEN variant = 'TREATMENT' THEN SNOWFLAKE.CORTEX.COMPLETE('claude-3-5-sonnet', 'Is a hot dog a sandwich?') ELSE SNOWFLAKE.CORTEX.COMPLETE('llama3.2-1b', 'Is a hot dog a sandwich?') END AS result FROM assignment; ``` ## ML Models This following examples demonstrate how you would use Winning Variant to run predictions for a set of customers using two model versions from the [Snowflake Model Registry](https://docs.snowflake.com/en/developer-guide/snowflake-ml/model-registry/overview). The experiment has the following definition: * A 50/50 test between control (`v1` of our model) and treatment (`v2`) * Winning Variant experiment ID of `ml-test` `customer_test_data` in these examples contains the following fields: * `customer_id`: ID of the customer * `age`: how long the customer has been a customer * `usage`: some arbitrary integer that defines how much the customer has used the product The sample model `customer_churn_model` has been trained to predict churn given an `age` and `usage` for a customer. The "Python (Simple)" example below uses the [Winning Variant Python SDK](/sdk/python). ```sql SQL theme={null} -- Make/get assignments for all customers IDs WITH assignments AS ( SELECT customer_id, create_assignment(customer_id, 'ml-test') AS variant FROM customer_test_data ), -- Pull model v1 (our CONTROL variant) model_v1 AS MODEL customer_churn_model VERSION v1, -- Pull model v2 (our TREATMENT variant) model_v2 AS MODEL customer_churn_model VERSION v2 -- If the assigned variant for the customer is 'TREATMENT', -- use v2, otherwise use v1 (CONTROL) SELECT c.customer_id, a.variant as experiment_variant, CASE WHEN a.variant = 'TREATMENT' THEN model_v2!predict(age, usage) ELSE model_v1!predict(age, usage) END AS prediction FROM customer_test_data c LEFT JOIN assignments a ON c.customer_id = a.customer_id; ``` ```python Python (Batch) theme={null} import snowflake.snowpark as snowpark from snowflake.snowpark.functions import call_udf, col from snowflake.ml.registry import Registry # Load the model from the registry registry = Registry(session) model = registry.get_model("customer_churn_model") # Get customer data to run predictions on df_customers = session.sql('SELECT * FROM customer_test_data') # Use the Winning Variant UDF to get/make assignments for customers customer_list = df_customers.with_column("experiment_variant", call_udf("create_assignment", col("customer_id"), "ml-test")).collect() # Split customers into experiment variants control_rows = [row for row in customer_list if row["EXPERIMENT_VARIANT"] == "CONTROL"] treatment_rows = [row for row in customer_list if row["EXPERIMENT_VARIANT"] == "TREATMENT"] # Convert the row lists back into DataFrames df_control = session.create_dataframe(control_rows) df_treatment = session.create_dataframe(treatment_rows) # Run inference v1_predictions = model.version('v1').run(df_control, function_name='predict') v2_predictions = model.version('v2').run(df_treatment, function_name='predict') # Show predictions v1_predictions.show() v2_predictions.show() ``` ```python Python (Single) theme={null} from snowflake.ml.registry import Registry from winningvariant import WinningVariantClient # Initialize the Winning Variant SDK wv = WinningVariantClient(session) # Load the model from the registry registry = Registry(session) model = registry.get_model("customer_churn_model") # Get the assignment for the user we want to run a prediction for assignment = wv.create_assignment(subject_id="user_123", experiment_id="my-exp") if assignment == "v2": prediction = model.version('v2').run(...) elif assignment == "v1": prediction = model.version('v1').run(...) else: # Something went wrong. Fall back to v1. prediction = model.version('v1').run(...) # Show prediction prediction.show() ``` # Use Case Walkthrus Source: https://docs.winningvariant.com/examples/use-case-walkthrus Full walkthurs of the setup and implementation of various use cases. ## Website Checkout A/B Test with Apple Pay Running experiments against e-commerce checkout flows is a common practice, used to improve the efficiency and effectiveness of the checkout process for users. Today, we walk through an example using Winning Variant on Snowflake AI Data Cloud. [Watch on YouTube](https://www.youtube.com/watch?v=1PdVQ2kaUxU) ## Split Testing ML Model Versions When developing a new ML model, before putting it out into the wild, it's important to test its performance against real business metrics. Not just historical testing (what might have happened if we had this version before), but in real time against live activity. Winning Variant allows you to run a split test comparing multiple model versions at the same time against live traffic. The results will provide an objective measure of change of the business metric we care about, and give a more reliable signal we can use to determine if the new model is "better" than the old. [Watch on YouTube](https://youtu.be/ttFCFWHTFrI) # Identifier Linking via API Source: https://docs.winningvariant.com/identifiers/api Linking identifiers via the RESTful API. ## Overview Use the API described below to link two different identifiers from within your application. The hostname for the API can be retrieved by clicking the App from within Snowflake: Data Products > Apps > Winning Variant Experimentation. The hostname for "Variant API" is the one you'll use. ## Identifier Link `POST /id-link` Use this endpoint to link two different identifiers. ### **Request Body** The request accepts a JSON-formatted body with the following parameters: | **Parameter** | **Type** | **Required** | **Description** | | :------------ | :------------ | :----------- | :-------------------------------------------------------------- | | `identifiers` | `Identifer[]` | Yes | List of identifiers to link together. Length must be exactly 2. | `Identifier` The identifer object in the body includes the following properties: | **Parameter** | **Types** | **Required** | **Description** | | -------------- | --------- | ------------ | ---------------------------------------------------- | | `subject_type` | `string` | Yes | The subject type defined in your app configuration. | | `subject_id` | `string` | Yes | ID for the subject that you want to link another to. | ### **Response** Upon success, returns a `200` status code with response `{ "success": true }`. ### Examples **Link an anonymous ID to a customer ID** In this example, we link an ID for subject type `ANONYMOUS_ID` to an ID for subject type `CUSTOMER_ID` (both subject types already exist). ``` POST /id-link Content-type: application/json Accept: application/json { "identifiers": [{ "subject_type": "CUSTOMER_ID", "subject_id": "ce0fb767-27de-477d-bebc-064dd724bb0b" }, { "subject_type": "ANONYMOUS_ID", "subject_id": "58865777-783b-4f41-9be4-a09afbaf187b" }] } Response: { "success": true } ``` ## Public Access To learn how you can access these services from outside of Snowflake, [read more here](https://docs.winningvariant.com/assignments/api#public-access). # Overview Source: https://docs.winningvariant.com/identifiers/overview About about Winning Variant identifier linking. Winning Variant allows you to explicitly link two different types of subject IDs to inform analytics processes that a metric set for one particular ID (such as a known user ID) can also be attributed to another ID (e.g., an anonymous ID). This is especially helpful when running A/B tests randomlized by an anonymous ID, but the resulting conversion you care about is stored against a customer ID. In this scenario, you would explicitly tell the app that the anonymous ID and customer ID are linked which would allow us to attribute outcomes to the proper variant exposure. ## Identifier Link Data [**Read more about the raw identifer link data**](/analytics/linked-identifiers) that’s passed through the application. # Identifier Linking via SQL Source: https://docs.winningvariant.com/identifiers/sql Linking identifiers using SQL ## Overview Identifiers can be linked directly in SQL. This is useful when when you have a real-time identifier match in a SQL process or when back-filling data. ## Roles A user must be assigned of the following [application roles](/snowflake-app-details#application-roles) to execute any of the examples described on this page: * `admin` * `editor` * `scientist` ## Link Identifiers To link two identifiers, simply make a stored procudure call. Note that the referenced Subject Types must exist already. ```SQL theme={null} SELECT experimentation.link_identifiers('', '', '', ''); ``` ## Examples ### Link an anonymous ID to a customer ID ```SQL theme={null} SELECT experimentation.link_identifiers('CUSTOMER_ID', 'u_123', 'ANON_ID', 'a_123'); ``` ### # Installation Source: https://docs.winningvariant.com/installation Install and configure the Winning Variant Snowflake Native App. The Winning Variant app is "by request", meaning that we must explicitly grant customers access to the app. If you're interested in what we have to offer, please click "Request" in the marketplace listing or [reach out directly](https://www.winningvariant.com/#get-started). ## 1. Set up an event table An event table is where all app logs will be saved. It's important to have this configured properly *before* installing the app to ensure logs are routed correctly. [Follow the instructions on Snowflake's site](https://docs.snowflake.com/en/developer-guide/logging-tracing/event-table-setting-up#create-an-event-table) ## 2. Install the app The app will need to be made available to your account prior to installation. Once available, you can find the app **Winning Variant Experimentation**. Click the "Get" button to begin installation. Installation may take a few minutes. When complete, you'll be able to see the app listed under **Data Products > Apps** in Snowsight. ## 3. Grant app permissions 1. Find the **Winning Variant** app under **Data Products > Apps** in Snowsight 2. Click the app to be taken to the activation screen. 3. Review the requested permissions listed on the screen. When you're ready to proceed, click **Grant**. 4. Once granted, an **Activate** button will appear -- click this to finalize setup. 5. After grants are finalized, you will be taken to the Winning Variant App Management screen. In order to view the App Management streamlit app, you must be the app owner or be granted a role mapped to the `ADMIN` application role. ## 4. Map roles Winning Variant creates a [set of application roles](/snowflake-app-details#application-roles) that can be used to manage access to the app's functionality. In order to use these, they must be mapped to roles that exist on your account. You can use an existing role or create a new one. The following example creates a new account role `experiment_editor` and maps it to the `editor` application role exposed by the app: ```sql theme={null} USE ; -- Create a new role on the account CREATE ROLE experiment_editor; -- Map this role to the Winning Variant `editor` role GRANT APPLICATION ROLE editor TO ROLE experiment_editor; ``` You can also map roles visually via the streamlit app by clicking into the app and clicking "Manage Access" in the top right of the page. ## 5. Wait for services to start Behind the scenes, the app is creating a compute pool, warehouses, and services. The status will be displayed on the streamlit dashboard. This process will take at least a few minutes. Refresh the page to monitor status changes. When all services are ready, all will turn green and show the public URLs for each service. Winning Variant Services Available ## 6. Dive in! Now that the Winning Variant native app is running, you can start using it immediately. ### Manage Experimentation Create labs, subject types, and experiments in your new experimentation engine. Learn about the different kinds of resources that can be managed. How to create and edit resources from within Snowflake. ### Experimention Data Learn about the raw experimentation data you can not activate for better, deeper analysis. What an assignment is and how you can use it. Learn how to access and query assignment data. # Introduction Source: https://docs.winningvariant.com/introduction Winning Variant: AI Impact Visibility Winning Variant allows customers to test incremental changes to their website, apps, backend services, and AI/ML - measuring the business impact of each. Deploy and analyze experiments faster with real time data available securely inside of Snowflake. Measure outcomes against existing business metrics in your current BI tool. The entire app runs in your Snowflake account, so no data moves in or out without your permission, ensuring a greater level of security. The end result is an AI Impact Visibility platform that looks and feels like it was built in-house, without you having to do so. ## Getting Started Getting up and running with Winning Variant is incredibly simple. We've broken down the steps to help you get going: 1. [Install the Snowflake Native App](/installation) 2. [Create and implement your first A/B test](/try-it-out) 3. Learn about [Assignments](/assignments/overview), which are the foundation for reporting ## Other helpful resources Once you're set up (or if you prefer to get into the nitty gritty sooner), here are some other articles we recommend you read: ### Snowflake Native App Setup Install and configure the Winning Variant Native App in your Snowflake account. Install and configure the Snowflake Native App. Learn more about what's under the hood of your newly installed app. ### Manage Experimentation Create labs, subject types, and experiments in your new experimentation engine. Learn about the different kinds of resources that can be managed. How to create and edit resources from within Snowflake. ### Experimention Data Learn about the raw experimentation data you can not activate for better, deeper analysis. What an assignment is and how you can use it. Learn how to access and query assignment data. # Experiments Source: https://docs.winningvariant.com/resources/experiments Any test you would like to run. An Experiment is the core abstraction used to assign variants to subjects in order to *experiment* (or test) a feature, model, idea, etc. An Experiment lives within a [Lab](/api/labs) and can have its lifecycle managed independently from any other active Experiments. ## Subjects An experiment must use exactly one [Subject Type](/resources/subjects). That is, you must define what type of identifier is used to assign and persist a variant across the life of the experiment. For example, if you have and use a `customer_id` subject type (one which identifies unique customers in some internal database), an experiment variant will be assigned to each unique `customer_id` encountered. When that `customer_id` is seen again in the future, it will receive the same variant it did previously for the experiment. (that is, until a winner is declared - [read about the experiment lifecycle](/experiment-lifecycle)). ## YAML ### Metadata When defining a resource in YAML, each must have a `metadata` section. The following properties within `metadata` are shared across all resource types: | Key | Data Type | Description | | ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `string` | Unique identifier. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `home-page-test`. IDs will be stored and displayed in UPPERCASE. | | `name` | `string` | Name of the resource. | | `description` | `string` | Description of the resource. | | `resourceVersion` | `int` | Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update. | | `status` | [`ResourceStatus`](#resourcestatus) | The status of the resource. | | `parentKind` | `lab` | The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments. | | `parentId` | `string` | The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.

If an experiment does not specified a `parentID`, it defaults to `DEFAULT`. | #### `ResourceStatus` Possible statuses include: | Status | Description | Resources | | ----------------- | ---------------------------------------------------------------- | ---------- | | `draft` | The experiment exists, but is not ready to go live. | Experiment | | `active` | The experiment is live and making/tracking/returned assignments. | All | | `winner_declared` | A winner has been declared. | Experiment | | `ended` | The experiment has ended. No assignments are made/returned. | Experiment | | `archived` | The experiment has ended and is archived from view. | All | ### Spec Experiments exhibit the following properties: | Key | Data Type | Description | | ---------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `subjectType` | `string` | ID of the [Subject](/resources/subjects) to use for this experiment. If not specified, will default to `ANY`, which allows any non-empty subject ID. | | `hypothesis` | `string` | An open field for stating the experiment hypothesis. | | `links` | `map` | Map of links to store as metadata for the experiment. Such as links to PRDs, analysis, etc. | | `variants` | `Variant[]` | List of variants. See below. | | `cohorts` | `Cohort[]` | List of cohorts. See below. | | `winningVariant` | `string` | ID of the winning variant, once status is changed to winner\_declared. | | `endedReason` | [`EndedReason`](#endedreason) | Why the experiment was ended. See below. There’s not practical use for this field, but rather is for record keeping. | #### `EndedReason` | Key | Description | | ------------------ | --------------------------------------------------------------------------- | | `success` | The experiment was a success, achieving statistical significance. | | `tech_issue` | There was a technical or implementation issue and the experiment was ended. | | `no_longer_needed` | The experiment is no longer needed or relevant. | | `no_stat_sig` | The experiment ran fine, but did not reach statistical significance. | | `other` | | #### `Variant` | Key | Data Type | Description | | ------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id` | `string` | Unique ID for the variant. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `treatment-a`. Variants will be stored and displayed in UPPERCASE. | | `isControl` | `bool` | Whether the variant is the control. Only one Variant may have this set per experiment. | | `name` | `string` | The name of the variant. | | `description` | `string` | Description of the variant. | #### `Cohort` | Key | Data Type | Description | | ----------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `index` | `int` | A unique integer auto-assigned to each cohort. This field is required and must be set to the next integer in sequence for her cohorts.

If the previous cohort was `4`, the new cohort must have value `5`. | | `variants` | [`CohortVariant[]`](#cohortvariant) | List of variants in this cohort with associated context. See below. | | `createdAt` | `timestamp` | When the cohort was created. This will be set automatically for new cohorts. Be sure to maintain the original values for existing/past cohorts when updating an experiment. | #### `CohortVariant` | Key | Data Type | Description | | --------- | --------- | ------------------------------------------------------------------------------------------------------- | | `variant` | `string` | ID of the Variant that’s active within the cohort. | | `split` | `float` | What percentage of traffic should see this variant. All split values within a Cohort must equal 1.000 . | ## Experiment Lifecycle An experiment may progress through different lifecycle phases in Winning Variant, each designed to handle variant assignments appropriately for subjects requesting one. The progression of an experiment is typically linear. That is, once it achieves a status, it typically does not go backwards and has a clear next status. Like any other resource within Winning Variant, an experiment has a status that denotes its current lifecycle stage. ### Draft An optional status, an experiment in “draft” simply means it is not ready to go live. This allows the entity to be created, but indicates that it is either not fully configured or is not ready to be made live (such as pending implementation). ### Active When an experiment is marked as “active”, it is recognized by the [Variant API](/api/variants) and has variants assigned and returned for subjects as requested. An experiment actively being evaluated should be in this state until it has been proven successful in some fashion. When a new assignment is requested, a variant will be randomly selected for the subject based on the split defined for that variant within the current cohort. For example, if a split is set to `0.500`, 50% of subjects will be assigned the variant. ### Winner Declared The “Winner Declared” status indicates that a “winning variant” has been chosen for the experiment. This could be due to statistical significance, preference in a tie, or any other reason. This status signals that *all* subjects should see the winning variant. When an experiment enters this status, all future assignment requests return the winning variant, regardless of past assignments. For example, if a subject was previously assigned the `control` variant, but `treatment-a` was declared the winner, the subject would be returned `treatment-a` on subsequent requests after the status change. This status should be used for experiments that are over, but are still depended upon by technical implementations. Once there are no dependencies on these experiment being active, it should be changed to “Ended”. ### Ended An experiment is “ended” when assignments no longer need to be served for it, but you still may want to reference reporting for it. It’s a sort of in-between status where it’s not used anywhere, but it’s also not archived, so it’s mostly for convenience in filtering/reporting. ### Archived Archive an experiment when it’s long gone, after reporting is no longer needed for it. Assignment data is never deleted, so it can be referenced in the future, but archiving old experiments helps keep your labs clean. ## Variant Assignments Variant assignments will be managed for any experiment in the `active` or `winner declared` status. In `active`, they will be assigned according to the `split` defined for that variant within the active cohort. If a subject is seen more than once, it will always receive the same variant it was assigned the first time. If in the `winner declared` status, all subjects will receive the winning variant, regardless of the original assignment. ## Examples ### Create draft experiment with one cohort ```yaml theme={null} schemaVersion: 1 kind: experiment metadata: id: hero-nov-2024 status: draft name: Hero text test for November 2024 description: Test variations of the hero on the website home page. Test launches in November 2024! parentKind: lab parentId: marketing spec: subjectType: anonymous_id hypothesis: "One of the new variants will perform better than the current static control." winningVariant: variants: - id: control isControl: true name: Control description: "Original, static title." - id: treatment-a isControl: false name: Limited Use case description: "Describes a limited use case to test specificity." cohorts: - index: 1 variants: - variant: control split: 0.5000 - variant: treatment-a split: 0.5000 ``` ### Add a cohort to the previous example **Cohort Index** Note the new cohort has an `index` of `2` since it's the next in the sequence. ```yaml theme={null} schemaVersion: 1 kind: experiment metadata: id: hero-nov-2024 status: draft name: Hero text test for November 2024 description: Test variations of the hero on the website home page. Test launches in November 2024! parentKind: lab parentId: marketing spec: subjectType: anonymous_id hypothesis: "One of the new variants will perform better than the current static control." winningVariant: variants: - id: control isControl: true name: Control description: "Original, static title." - id: treatment-a isControl: false name: Limited Use case description: "Describes a limited use case to test specificity." cohorts: - index: 1 createdAt: 2020-08-22T04:30:08Z variants: - variant: control split: 0.5000 - variant: treatment-a split: 0.5000 - index: 2 variants: - variant: control split: 0.7500 - variant: treatment-a split: 0.2500 ``` ### Active experiment with multiple cohorts ```yaml theme={null} schemaVersion: 1 kind: experiment metadata: id: hero-nov-2024 resourceVersion: 1 # Ignored on update status: active name: Hero text test for November 2024 description: Test variations of the hero on the website home page. Test launches in November 2024! parentKind: lab parentId: marketing spec: subjectType: anonymous_id hypothesis: "One of the new variants will perform better than the current static control." winningVariant: variants: - id: control isControl: true name: Control description: "Original, static title." - id: treatment-a isControl: false name: Limited Use case description: "Describes a limited use case to test specificity." - id: treatment-b isControl: false name: Expanded Use case description: "Describes a lengthier use case to test generality." cohorts: - index: 1 createdAt: 2020-08-22T04:30:08Z variants: - variant: control split: 0.5000 - variant: treatment-a split: 0.5000 - index: 2 createdAt: 2020-09-22T04:30:08Z variants: - variant: control split: 0.3333 - variant: treatment-a split: 0.3333 - variant: treatment-b split: 0.3333 ``` ### Experiment with winner declared **Winning Variant** Must be the ID of a variant defined within the experiment. ```yaml theme={null} schemaVersion: 1 kind: experiment metadata: id: hero-nov-2024 resourceVersion: 1 # Ignored on update status: winner_declared name: Hero text test for November 2024 description: Test variations of the hero on the website home page. Test launches in November 2024! parentKind: lab parentId: marketing spec: subjectType: anonymous_id hypothesis: "One of the new variants will perform better than the current static control." winningVariant: treatment-a variants: - id: control isControl: true name: Control description: "Original, static title." - id: treatment-a isControl: false name: Limited Use case description: "Describes a limited use case to test specificity." - id: treatment-b isControl: false name: Expanded Use case description: "Describes a lengthier use case to test generality." cohorts: - index: 1 createdAt: 2020-08-22T04:30:08Z variants: - variant: control split: 0.5000 - variant: treatment-a split: 0.5000 - index: 2 createdAt: 2020-09-22T04:30:08Z variants: - variant: control split: 0.3333 - variant: treatment-a split: 0.3333 - variant: treatment-b split: 0.3333 ``` # Labs Source: https://docs.winningvariant.com/resources/labs Logical groupings for experiments. A Lab is simply a logical grouping of experiment resources. In the future, other potential resource assets may be group-able into Labs, but for now, think of a Lab as a folder or workspace that experiments live inside. Common use cases for labs include: * by org (i.e., Marketing, Engineering) * by team * used as folders There are no limits to the number of labs that can be created. Go crazy! ## YAML ### Metadata The possible statuses for labs are: `active` or `archived`. `parentKind` and `parentId` should be excluded. When defining a resource in YAML, each must have a `metadata` section. The following properties within `metadata` are shared across all resource types: | Key | Data Type | Description | | ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `string` | Unique identifier. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `home-page-test`. IDs will be stored and displayed in UPPERCASE. | | `name` | `string` | Name of the resource. | | `description` | `string` | Description of the resource. | | `resourceVersion` | `int` | Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update. | | `status` | [`ResourceStatus`](#resourcestatus) | The status of the resource. | | `parentKind` | `lab` | The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments. | | `parentId` | `string` | The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.

If an experiment does not specified a `parentID`, it defaults to `DEFAULT`. | #### `ResourceStatus` Possible statuses include: | Status | Description | Resources | | ----------------- | ---------------------------------------------------------------- | ---------- | | `draft` | The experiment exists, but is not ready to go live. | Experiment | | `active` | The experiment is live and making/tracking/returned assignments. | All | | `winner_declared` | A winner has been declared. | Experiment | | `ended` | The experiment has ended. No assignments are made/returned. | Experiment | | `archived` | The experiment has ended and is archived from view. | All | ### Spec Labs don’t have any special that needs to be defined that isn’t in the metadata, so no `spec` is necessary. ### Example ```yaml theme={null} schemaVersion: 1 kind: lab metadata: id: marketing resourceVersion: 1 status: active name: Marketing description: All initiatives for the marketing org. ``` # Manage Source: https://docs.winningvariant.com/resources/manage Configure Winning Variant resources directly in Snowflake The Winning Variant Native App allows users to manage experimentation resources directly inside of Snowflake by use of stored procedures created during installation. ## Roles A user must be assigned of the following [application roles](/snowflake-app-details#application-roles) to execute any of the procedures described on this page: * `admin` * `editor` ## Resource Kinds All references to `kind` below must be one of: * `lab` * `subjecttype` * `experiment` [Read more about Resource Kinds](/resources/overview#resource-kinds). ## List Resources Each of the following "list" commands returns a table with the following columns: | Column | Description | Data Type | | ------------------ | ------------------------ | --------- | | `kind` | The resource kind. | `VARCHAR` | | `id` | ID of the resource. | `VARCHAR` | | `name` | Name of the resource. | `VARCHAR` | | `status` | Resource status | `VARCHAR` | | `description` | Resource description | `VARCHAR` | | `resource_version` | Current resource version | `INT` | ```SQL theme={null} USE ; CALL experimentation.list_resources('' [, , ...]); ``` Returns a list of all resources of the given kind. `status` must be one of [ResourceStatus](/resources/overview#resourcestatus). ### Examples ```SQL Labs theme={null} -- List all USE ; CALL experimentation.list_resources('lab'); -- List active CALL experimentation.list_resources('lab', ['active']); ``` ```SQL Subject Types theme={null} USE ; CALL experimentation.list_resources('subjecttype'); -- List active CALL experimentation.list_resources('subjecttype', ['active']); ``` ```SQL Experiments theme={null} USE ; CALL experimentation.list_resources('experiment'); -- List active CALL experimentation.list_resources('experiment', ['active']); -- List active or winner declared CALL experimentation.list_resources('experiment', ['active','winner_declared']); ``` ## Get YAML for a single resource ```SQL theme={null} USE ; CALL experimentation.get_resource_yaml('', ''); ``` Returns a table with the following columns: | Column | Description | Data Type | | ------ | ---------------------------------------------------- | --------- | | `kind` | The resource kind. | `VARCHAR` | | `id` | ID of the resource. | `VARCHAR` | | `yaml` | The full configuration (YAML) for a single resource. | `VARCHAR` | ### Examples ```SQL theme={null} USE ; CALL experimentation.get_resource_yaml('experiment', 'my-experiment'); ``` ## Create or update resource ```SQL theme={null} USE ; CALL experimentation.apply_resource(''); ``` If the resource with the `id` specified exists, it gets updated, otherwise it will be created. ### Examples ```SQL theme={null} USE ; -- Create a new lab with id `marketing` CALL experimentation.apply_resource($$schemaVersion: 1 kind: lab metadata: id: marketing status: active name: Marketing description: All marketing and top-of-funnel experiments. $$); -- Update the name of the lab we just created CALL experimentation.apply_resource($$schemaVersion: 1 kind: lab metadata: id: marketing status: active name: Marketing Organization description: All marketing and top-of-funnel experiments. $$); ``` ## Experimentation Convenience Methods In addition to the above methods using raw YAML, you may also create/manage experiments with the following purpose-built convenience methods. All methods also have a `*_preview` variant is available that returns the YAML that would be applied without actually creating the experiment: ```sql theme={null} CALL experimentation.create_experiment_preview( 'homepage-cta-test', 'Homepage CTA Test', NULL, NULL, NULL, 'USER', '[{"id":"control","name":"Blue Button","isControl":true}]', NULL ); ``` ### Create Experiment Creates a new experiment with the following inputs. The experiment is created in `draft` status. | Parameter | Description | Data Type | Example | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------- | | `id` | Unique identifier for the experiment. Alphanumeric, hyphens, underscores, dots only. | `VARCHAR` | `'homepage-cta-test'` | | `name` | Human-readable name for the experiment. | `VARCHAR` | `'Homepage CTA Test'` | | `description` | Optional description of the experiment. Pass `NULL` to omit. | `VARCHAR` | `'Testing button color on homepage'` | | `parent_lab` | Optional ID of the parent lab. Pass `NULL` to use the DEFAULT lab. Must exist if provided. | `VARCHAR` | `'growth-lab'` | | `hypothesis` | Optional hypothesis for the experiment. Pass `NULL` to omit. | `VARCHAR` | `'A green CTA will increase conversions by 10%'` | | `subject_type` | ID of the subject type. Must already exist. | `VARCHAR` | `'USER'` | | `variants` | JSON array of variant objects. Each must have `id`, `name`, and `isControl`. Optional `description`. At most 1 control. | `VARCHAR` (JSON) | `'[{"id":"control","name":"Blue Button","isControl":true},{"id":"green","name":"Green Button","isControl":false}]'` | | `initial_cohort` | JSON array of cohort variant entries with `variant` and `split`. Splits must sum to 1.0. Requires `variants`. Pass `NULL` to omit. | `VARCHAR` (JSON) | `'[{"variant":"control","split":0.5},{"variant":"green","split":0.5}]'` | **Returns:** Table with columns `operation`, `kind`, `id`, `name`, `status`, `description`. ```sql theme={null} CALL experimentation.create_experiment( 'homepage-cta-test', 'Homepage CTA Test', 'Testing button color on homepage', NULL, 'A green CTA will increase conversions by 10%', 'USER', '[{"id":"control","name":"Blue Button","isControl":true},{"id":"green","name":"Green Button","isControl":false}]', '[{"variant":"control","split":0.5},{"variant":"green","split":0.5}]' ); ``` ### Add Variants Adds one or more new variants to an existing experiment. Variant IDs must not conflict with existing variants. At most 1 control variant is allowed across all variants on the experiment. | Parameter | Description | Data Type | Example | | --------------- | ---------------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------ | | `experiment_id` | ID of the experiment to add variants to. Must exist. | `VARCHAR` | `'homepage-cta-test'` | | `variants` | JSON array of variant objects. Each must have `id`, `name`, and `isControl`. Optional `description`. | `VARCHAR` (JSON) | `'[{"id":"red","name":"Red Button","isControl":false,"description":"Red variant"}]'` | **Returns:** Table with columns `operation`, `kind`, `id`, `name`, `status`, `description`. ```sql theme={null} CALL experimentation.add_variants( 'homepage-cta-test', '[{"id":"red","name":"Red Button","isControl":false,"description":"A red CTA button"}]' ); ``` ### Add Cohort Adds a new cohort to an existing experiment. The cohort index is automatically assigned as the next sequential value. The experiment must already have variants defined. | Parameter | Description | Data Type | Example | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------- | | `experiment_id` | ID of the experiment. Must exist and have variants defined. | `VARCHAR` | `'homepage-cta-test'` | | `cohort_variants` | JSON array of cohort entries. Each must have `variant` (must be an existing variant on the experiment) and `split` (0.001–1.0). Splits must sum to 1.0. At most 1 control variant. | `VARCHAR` (JSON) | `'[{"variant":"control","split":0.34},{"variant":"green","split":0.33},{"variant":"red","split":0.33}]'` | **Returns:** Table with columns `operation`, `kind`, `id`, `name`, `status`, `description`. ```sql theme={null} CALL experimentation.add_cohort( 'homepage-cta-test', '[{"variant":"control","split":0.34},{"variant":"green","split":0.33},{"variant":"red","split":0.33}]' ); ``` ### Set Experiment Status Sets the status of an existing experiment. | Parameter | Description | Data Type | Example | | --------------- | -------------------------------------------------------------------------------------- | --------- | --------------------- | | `experiment_id` | ID of the experiment. Must exist. | `VARCHAR` | `'homepage-cta-test'` | | `status` | New status. Must be one of: `active`, `archived`, `draft`, `ended`, `winner_declared`. | `VARCHAR` | `'active'` | **Returns:** Table with columns `operation`, `kind`, `id`, `name`, `status`, `description`. ```sql theme={null} CALL experimentation.set_experiment_status('homepage-cta-test', 'active'); ``` ### Update Experiment Updates one or more properties of an existing experiment. Pass `NULL` for any field that should not change. At least one field must be provided. | Parameter | Description | Data Type | Example | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ---------------------------------- | | `experiment_id` | ID of the experiment. Must exist. | `VARCHAR` | `'homepage-cta-test'` | | `name` | New name. Pass `NULL` to keep current. | `VARCHAR` | `'Updated CTA Test'` | | `description` | New description. Pass `NULL` to keep current. | `VARCHAR` | `'Revised experiment description'` | | `status` | New status. Must be one of: `active`, `archived`, `draft`, `ended`, `winner_declared`. Pass `NULL` to keep current. | `VARCHAR` | `'ended'` | | `hypothesis` | New hypothesis. Pass `NULL` to keep current. | `VARCHAR` | `'Updated hypothesis'` | | `ended_reason` | Reason the experiment ended. Only valid when status is `ended`. Must be one of: `no_longer_needed`, `no_stat_sig`, `other_reason`, `success`, `tech_issue`. | `VARCHAR` | `'success'` | | `winning_variant` | ID of the winning variant. Only valid when status is `winner_declared`. Must be a variant on the experiment. | `VARCHAR` | `'green'` | **Returns:** Table with columns `operation`, `kind`, `id`, `name`, `status`, `description`. ```sql theme={null} -- Update just the name and hypothesis CALL experimentation.update_experiment( 'homepage-cta-test', 'New Name', NULL, NULL, 'New hypothesis', NULL, NULL ); -- Declare a winner CALL experimentation.update_experiment( 'homepage-cta-test', NULL, NULL, 'winner_declared', NULL, NULL, 'green' ); -- End an experiment with a reason CALL experimentation.update_experiment( 'homepage-cta-test', NULL, NULL, 'ended', NULL, 'no_stat_sig', NULL ); ``` ### Get Current Cohort Returns the current (highest-indexed) cohort for an experiment, with variant details. | Parameter | Description | Data Type | Example | | --------------- | -------------------------------------------------------------- | --------- | --------------------- | | `experiment_id` | ID of the experiment. Must exist and have at least one cohort. | `VARCHAR` | `'homepage-cta-test'` | **Returns:** Table with columns `cohort_index` (NUMBER), `variant` (VARCHAR), `name` (VARCHAR), `description` (VARCHAR), `is_control` (BOOLEAN), `split` (FLOAT). ```sql theme={null} CALL experimentation.get_current_cohort('homepage-cta-test'); ``` ### Describe Experiment YAML Given experiment YAML, returns a markdown-formatted description of the experiment including metadata, variants, and cohort details. | Parameter | Description | Data Type | Example | | --------- | ---------------------------- | --------- | ----------------------------------- | | `yaml` | Full experiment YAML string. | `VARCHAR` | *`(output from get_resource_yaml)`* | **Returns:** Table with a single `markdown` column containing the formatted description. ```sql theme={null} -- First get the YAML, then describe it CALL experimentation.get_resource_yaml('EXPERIMENT', 'homepage-cta-test'); -- Use the YAML output as input: CALL experimentation.describe_experiment_yaml(''); ``` # Manage Experiments with AI Source: https://docs.winningvariant.com/resources/manage-experiments-with-ai Use Snowflake Intelligence to create and manage experiments. Use Snowflake Intelligence and Cortex to manage experiments. Winning Variant ships with a "Winning Variant Experiment Assistant" Cortex agent (in the `ai` schema) that allows you to create and manage experiments via Snowflake Intelligence. Simply describe what you would like to do to have the assets created or updated for you. ## Creating Experiments Describe the experiment you would like to create. Include as many details as possible. The agent will prompt you for more information if needed. Experiment properties include: * Name * Subject Type * Description (optional) * Hypothesis (optional) * Lab (optional) * Variants in the experiment, with names, descriptions, and which is control * Initial cohort: the split for each variant Create Experiment ## Listing / Detailing Experiments List experiments of a particular status or get details for an experiment. Sample prompts include: * *List all active experiments* * *List experiments with a winner declared* * *Give me the details of XYZ experiment* * *Show the current cohort for XYZ experiment* List Experiment ## Adding Variants & Cohorts Variants and cohorts may be added over the life of an experiment. Sample prompts: * *Add a new variant "Treatment B" to experiment XYZ with the description "Slight modification to experiment"* * *Add a new cohort for XYZ test: control 20%, treatment A 40%, treatment B 40%* * *Ramp up XYZ to 50/50* New Cohort ## Changing Experiment Status The [experiment lifecycle](/resources/experiments) can be managed, as well. Simply describe what status you want to move the experiment to with associated information. Sample prompts: * *Set the treatment as the winner for XYZ experiment* * *Activate XYZ experiment* * *Mark XYZ experiment as ended* Winner Declared All recordings above have been sped up through the Cortex thinking stage. Actual speed of responses may vary. # Overview Source: https://docs.winningvariant.com/resources/overview Winning Variant implements a small set of first order resources that can be managed via the Configuration API. ## Managing Resources Resources are all managed within Snowflake via stored procedures within the Winning Variant Application. [Read more here](/resources/manage) ## Resource Kinds * **[Labs](/resources/labs) (Key: `lab`)**: Logical grouping of experiments. You can think of this as a workspace or even just as a folder containing experiments. Each experiment belongs to a lab. * **[Subject Types](/resources/subjects) (Key: `subjecttype`)**: Used by experiments, these are types of identifiers that are assigned variants within an experiment. Subject types exist at the account level and can be used across all experiments within all labs. Each experiment must specify the subject type to use for assignments. * **[Experiments](/resources/experiments) (Key: `experiment`)**: Any type of experiment you wish to run. ## YAML ### Schema Each resource requires a top-level `schemaVersion`, a resource `kind`, and a `metadata` object. Subject types and experiments also require a `spec` section with details on the resource. For example: ```yaml theme={null} schemaVersion: 1 kind: lab | subjecttype | experiment metadata: spec: ``` ### Metadata When defining a resource in YAML, each must have a `metadata` section. The following properties within `metadata` are shared across all resource types: | Key | Data Type | Description | | ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `string` | Unique identifier. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `home-page-test`. IDs will be stored and displayed in UPPERCASE. | | `name` | `string` | Name of the resource. | | `description` | `string` | Description of the resource. | | `resourceVersion` | `int` | Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update. | | `status` | [`ResourceStatus`](#resourcestatus) | The status of the resource. | | `parentKind` | `lab` | The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments. | | `parentId` | `string` | The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.

If an experiment does not specified a `parentID`, it defaults to `DEFAULT`. | #### `ResourceStatus` Possible statuses include: | Status | Description | Resources | | ----------------- | ---------------------------------------------------------------- | ---------- | | `draft` | The experiment exists, but is not ready to go live. | Experiment | | `active` | The experiment is live and making/tracking/returned assignments. | All | | `winner_declared` | A winner has been declared. | Experiment | | `ended` | The experiment has ended. No assignments are made/returned. | Experiment | | `archived` | The experiment has ended and is archived from view. | All | ### Spec This is specific to the resource kind. View the pages for [Labs](/resources/labs), [Subjects](/resources/subjects), or [Experiments](/resources/experiments) for more info. ### Examples ```yaml Lab theme={null} schemaVersion: 1 kind: lab metadata: id: marketing resourceVersion: 1 status: active name: Marketing description: All initiatives for the marketing org. ``` ```yaml Subject Type theme={null} schemaVersion: 1 kind: subjecttype metadata: id: anonymous-id resourceVersion: 2 status: active name: Anonymous ID description: Public, anonymous ID for a user (such as the cookie ID). spec: ... ``` ```yaml Experiment theme={null} schemaVersion: 1 kind: experiment metadata: id: hero-nov-2024 resourceVersion: 1 status: draft name: Hero text test for November 2024 description: Test variations of the hero on the website home page. Test launches in November 2024! parentKind: lab parentId: marketing spec: ... ``` # Subjects Source: https://docs.winningvariant.com/resources/subjects Identifiers that may be used in experiments. Winning Variant considers a subject anything that is assigned a variant within an experiment. Think of a subject as a patient in a drug trial. Traditionally, a subject is a user (known or anonymous), but can be any other *thing* that seems one thing over another, such as a mobile or IoT device. Subjects can have various ways of being identified (such as a known customer ID or an anonymous cookie ID). We call these *Subject Types* and are completely customizable within the platform. Here are some examples of typical Subject Types: * **Anonymous User ID**: such as a random UUID stored in a cookie in a user’s browser * **User/Customer ID**: a persistent ID that tells you exactly who someone is. Values used here are sometimes the CRM ID (such as Salesforce ID) or primary key in your own database. * **MAC Address**: when experiment assignments are done at the device level. * **Customer Group**: if you wish for all users within a particular customer group/segment to experiment the same experiment variation, you can use this level of grouping. All possible Subject Types can exist at once within the platform and are used only for the experiments that depend on them. ## YAML ### Metadata The possible statuses for subject types are: `active` or `archived`. `parentKind` and `parentId` should be excluded. When defining a resource in YAML, each must have a `metadata` section. The following properties within `metadata` are shared across all resource types: | Key | Data Type | Description | | ----------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `string` | Unique identifier. Case insensitive and consist of alphanumeric characters, '-', '\_' or '.’ For example: `home-page-test`. IDs will be stored and displayed in UPPERCASE. | | `name` | `string` | Name of the resource. | | `description` | `string` | Description of the resource. | | `resourceVersion` | `int` | Modification version for a given resource. Incremented each time the resource is updated. This will be shown when retreiving a resource, but is ignored if provided as part of a resource update. | | `status` | [`ResourceStatus`](#resourcestatus) | The status of the resource. | | `parentKind` | `lab` | The kind of resouce this one belongs to. Only applies to experiments, in which case the value is `lab`. Exclude for non experiments. | | `parentId` | `string` | The ID of the resources parent. Only applies to experiments and the ID of its parent lab. Exclude for non experiments.

If an experiment does not specified a `parentID`, it defaults to `DEFAULT`. | #### `ResourceStatus` Possible statuses include: | Status | Description | Resources | | ----------------- | ---------------------------------------------------------------- | ---------- | | `draft` | The experiment exists, but is not ready to go live. | Experiment | | `active` | The experiment is live and making/tracking/returned assignments. | All | | `winner_declared` | A winner has been declared. | Experiment | | `ended` | The experiment has ended. No assignments are made/returned. | Experiment | | `archived` | The experiment has ended and is archived from view. | All | ### Spec The `spec` for a Subject Type includes the following: | Key | Data Type | Description | | ------------ | ------------------------- | --------------------------------------------------------------------------------- | | `matchType` | [`MatchType`](#matchtype) | The type of subject and how to match identifiers. See below. | | `matchRegex` | `string` | If Match Type is CUSTOM, this specifies the Regex to use to validate identifiers. | #### `MatchType` Built-in Subject Type match types include: | Match Type | Description | Regex Used | | ------------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `UUID` | a standard UUID of any version | `^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-5][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$` | | `MAC_ID` | A computer MAC address | `^([0-9A-Fa-f]{2}[:]){5}([0-9A-Fa-f]{2})$` | | `KSUID` | Segment [KSUID](https://github.com/segmentio/ksuid) | `^[0-9a-zA-Z]{27}$` | | `IOS_INSTALLATION_ID` | iOS Installation ID | `^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$` | | `ANDROID_INSTALLATION_ID` | Android Installation ID | `^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$` | | `AUTH0_ID` | ID for [Auth0](https://auth0.com/) user accounts. | `^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89abAB][0-9a-f]{3}-[0-9a-f]{12}$` | | `SEGMENT_ID` | [Segment](https://segment.com/) user IDs | `^[0-9a-f]{32}$` | | `CUSTOM` | Custom regular expression validation. | Specified by `match_regex` | ### Examples ```yaml UUID Type theme={null} schemaVersion: 1 kind: subjecttype metadata: id: customer_id status: active name: Customer ID description: Internal customer ID for a user. spec: matchType: uuid ``` ```yaml Custom theme={null} schemaVersion: 1 kind: subjecttype metadata: id: employee_id status: active name: Dunder Mifflin employee ID description: Internal identifier used to identify Dunder Mifflin employees. spec: matchType: custom matchRegex: "^[0-9]{6}$" ``` # Python SDK Source: https://docs.winningvariant.com/sdk/python Run split tests with less friction directly in Python. The Winning Variant Python SDK is currently in Public Preview. If you notice any issues with the SDK, please report issues to your Winning Variant account team. The Winning Variant Python SDK makes it easier to implement split tests directly in your python applicatios, AI/ML workloads, or Streamlit apps. It uses an existing Winning Variant Snowflake Native App installation, accessed via a `session` object provided to the client. ## Installation ### General Installation Install the SDK by running the following: `pip install winningvariant` ### Import into Snowflake UDFs and procedures using Artifact Registry The [Snowflake Artifact Registry](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages#artifact-repository-overview) allows you to directly use Python packages from the Python Package Index (PyPI) within Snowpark Python user-defined functions (UDFs) and stored procedures. Follow the instructions in the linked article to use the `winningvariant` package. ### Use in Snowflake Notebooks / Worksheets Within a notebook or worksheet, select **Packages** and add the `winningvariant` package. Python Sdk In Snowflake ## Initialization To initialize the SDK, import the object and inintialize it using a Snowflake Snowpark `session` object: ```python theme={null} from winningvariant import WinningVariantClient from snowflake.snowpark import Session connection_parameters = { "user": "", "password": "", "account": "", "warehouse": "", "database": "" } session = Session.builder.configs(connection_parameters).create() wv = WinningVariantClient(session) ``` ### Options The following intializing parameters are available: | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------- | | `session` | The `snowflake.snowpark` session to use to access the Winning Variant Snowflake Native App. | | `cache` | (Default: `True`) If `True`, enables local caching of assignments as they're read/created. | The user defined in the `session` must have one of the following [application roles](https://docs.winningvariant.com/snowflake-app-details#application-roles): * `admin` * `editor` * `scientist` **Caching** Caching can be enabled/disabled after client initialization by calling `wv.enable_cache()` or `wv.disable_cache()`, respectively. ## Assignment Object The SDK includes an `Assignment` object that is used to identify the ID of a variant that a subject is assigned within an experiment. [Read more here](/assignments/overview). The Assignment object includes a reference to the `subject_id`, `experiment_id`, and the `variant` it's assigned to. ### Assignment Comparison To test if an assignment is for a given variant, you can call `is_variant("")` or do a string comparison: ```python theme={null} if assignment.is_variant("control"): print("Control group") if assignment == "control": print("Also control group") ``` ## Assignment Management The SDK provides various ways to get/create experiment assignments based on your needs. ### Get Assignment This read-only method gets an existing assignment for a subject ID inside of an experiment. If none exists, it will **not** be created. ```python theme={null} .get_assignment(subject_id="...", experiment_id="...") ``` | **Parameter** | **Description** | **Type** | | --------------- | -------------------------------------------------------- | -------- | | `subject_id` | (**Required**) ID of the subject used in the experiment. | `string` | | `experiment_id` | (**Required**) ID of the experiment. | `string` | Returns an [Assignment](#assignment-object) object if one exists, otherwise `None`. ```python Example theme={null} assignment = wv.get_assignment(subject_id="user_123", experiment_id="my-exp") if assignment == None: print("No assignment for this subject") elif assignment == "TREATMENT": print("Subject is in the treatment group") elif assignment == "CONTROL": print("Subject is in the control group") ``` ### Get or Create Assignment Gets an assignment if one exists, otherwise it creates a new one according to the experiment definition. ```python theme={null} .create_assignment(subject_id="...", experiment_id="...") ``` | **Parameter** | **Description** | **Type** | | :-------------- | :------------------------------------------------------- | :------- | | `subject_id` | (**Required**) ID of the subject used in the experiment. | `string` | | `experiment_id` | (**Required**) ID of the experiment. | `string` | Returns an [Assignment](#assignment-object) object upon success, `None` if something went wrong. ```python Example theme={null} assignment = wv.create_assignment(subject_id="user_123", experiment_id="my-exp") if assignment == "TREATMENT": print("Subject is in the treatment group") elif assignment == "CONTROL": print("Subject is in the control group") else: print("Something went wrong") ``` ### Check if a subject has a specific assignment with an experiment If you have a subject for which you want to do a quick check to see if they have a particular assignment or not, you can use the shorthand `check_variant` function. A good use case for this would be in 2-variant A/B tests or feature flag scenarios where you want to quickly check if a subject is in the treatment group. ```python theme={null} .check_variant(subject_id="...", experiment_id="...", variant_id="...") ``` | **Parameter** | **Description** | **Type** | | :------------------ | :------------------------------------------------------------------------------ | :------- | | `subject_id` | (**Required**) ID of the subject used in the experiment. | `string` | | `experiment_id` | (**Required**) ID of the experiment. | `string` | | `variant_id` | (\*\*Required) \*\*ID of the variant to check. | `string` | | `create_assignment` | (Default: True) If set, an assignment wll be made if one doesn't already exist. | `bool` | Returns a boolean indicating if the subject has the given assignment in the experiment. ```python Example theme={null} is_treatment = wv.check_variant(subject_id="user_123", experiment_id="my-exp", variant_id="treatment") if is_treatment: print("In the treatment group") else: print("Not in the treatment group") ``` ## Decorators The SDK provides a number of function decorators to provide optionality in how you implement split tests in our codebase. Each decorator requires the following: 1. An explicit `subject_id` *OR* `subject_arg` that specifes the argument passed to the function that will include the subject ID. 2. An explicit `experiment_id` *OR* `experiment_arg` that specifies the argument passed to the function that will include the experiment ID. ### `@.assignment` This decorator provides the assignment to the wrapped function. If no assignment exist for the subject, one is created. Supports syncronous and asyncronous functions. ```python Example theme={null} @wv.assignment(subject_arg="subject_id", experiment_arg="experiment_id") def my_func(subject_id, experiment_id, assignment = None): return f"Subject {subject_id} assigned to {assignment.variant}" my_func(subject_id="user_123", experiment_id="my-exp") ``` ### `@.if_assignment` This decorator executes the wrapped function only if the subject has the given assignment within the experiment. In addition to the general required parameter combinations defined above, this decorator has a few additional parameters: | **Parameter** | **Description** | **Type** | | ------------------- | -------------------------------------------------------------------- | -------- | | `variant_id` | (**Required**) The ID of the variant to compare against. | `string` | | `create_assignment` | (Default: True) If True, creates an assignment if one doesn't exist. | `bool` | ```python Example theme={null} @wv.if_assignment(subject_arg="subject_id", experiment_arg="experiment_id", variant_id="treatment") def my_func(subject_id, experiment_id): print("Subject is assigned to the treatment variant") # The function will only be run if `user_123` is assigned to the `treatment` variant my_func(subject_id="user_123", experiment_id="my-exp") ``` ### `@.unless_assignment` This decorator executes the wrapped function only if the subject **DOES NOT HAVE** the given assignment within the experiment. In addition to the general required parameter combinations defined above, this decorator has a few additional parameters: | **Parameter** | **Description** | **Type** | | :------------------ | :------------------------------------------------------------------- | :------- | | `variant_id` | (**Required**) The ID of the variant to compare against. | `string` | | `create_assignment` | (Default: True) If True, creates an assignment if one doesn't exist. | `bool` | ```python Example theme={null} @wv.unless_assignment(subject_arg="subject_id", experiment_arg="experiment_id", variant_id="treatment") def my_func(subject_id, experiment_id): print("Subject is NOT assigned to the treatment variant") # The function will only be run if `user_123` is NOT assigned to the `treatment` variant my_func(subject_id="user_123", experiment_id="my-exp") ``` # Try it out Source: https://docs.winningvariant.com/try-it-out Quickstart to trying out the Winning Variant Snowflake Native App. Winning Variant is designed to be simple to get started with. While there are many ways to [configure and manage](/advanced/app-configuration) the app or [customize your experiments](/resources/overview), you can get started creating your first experiment right away. The Snowflake Native App manages its own infrastructure internally, so there's nothing else you need to do once it's installed and running. ## Create an A/B test (experiment) To create your first experiment, simply pass the [experiment configuration](/resources/experiments) (via YAML) to a built-in stored procedure `apply_resource`. ```sql theme={null} CALL experimentation.apply_resource($$schemaVersion: 1 kind: experiment metadata: id: my-first-experiment status: active name: My First Experiment spec: variants: - id: control isControl: true name: Control - id: treatment isControl: false name: Treatment cohorts: - index: 1 variants: - variant: control split: 0.5 - variant: treatment split: 0.5 $$); ``` This creates an experiment named "My First Experiment" (ID: `my-first-experiment`). It has two variants (`control` and `treatment`), each receiving 50% of traffic in a single cohort. ## Make Assignments Every experiment needs to have some identifier for which traffic is randomized. Typically this is a user ID, anonymous ID, session ID, etc. We call this a [Subject](/resources/subjects). An assignment is the variant that a subject ID is *assigned* within a given experiment. Once assigned, a subject ID will be pinned to that variant forever and always. There are currently 3 ways to implement experiments and to get/set assignments: 1. Directly in [SQL](/assignments/sql) 2. Via the [RESTful API](/assignments/api) 3. Using the [Python SDK](/sdk/python) Between these 3 methods, it's possible to implement any type of split test in any environment, inside or outside of Snowflake. For the sake of example, we'll create an assignment using SQL: ``` SELECT experimentation.create_assignment('user_123', 'my-first-experiment'); SELECT experimentation.create_assignment('user_456', 'my-first-experiment'); ``` This creates an assignment for each of two users in our new experiment. If you call this same commands over and over, you'll see that the same variant is returned each time for the respective users. [See other implementation examples](/examples/use-case-walkthrus) ## View the data The internal assignment data pipeline is typically delayed 30-60 minutes ([this is configurable](/advanced/app-configuration)), so you'll need to set a timer and check back. When you do, you can query start to query the raw data, which will become the foundation for your experimentation reporting. ### Experiments You can view the experiment we created: ``` SELECT * FROM experimentation.experiments; ``` .. or the variants in the experiment: ``` SELECT * FROM experimentation.variants; ``` ... or the cohorts in the experiment: ``` SELECT * FROM experimentation.cohorts; ``` [Read more about the dimension tables available](/analytics/dimensions) ### Assignments You can also view the assignments you created above: ``` SELECT * FROM experimentation.assignments; ``` [Read more about analyzing assignments](/analytics/assignments) ## You did it! That's the basic setup and implementation of split tests using the Winning Variant Snowflake Native App. At this point, you have everything in place to run live experiments and analyize the results. You may be asking, ***What about the reports?*** Well, we *intentionally* do not include reporting in the product. Companies often have nuanced metrics they care about and shouldn't be forced to consume overly-opinionated dashboards that they have to log into separately. Instead, we provide access to raw experimentation data and are available to help your team create a custom analytics pipeline to ultimately render experiment results in your existing BI tool (e.g., Snowflake Dashboards, Tableau, Sigma, etc).