Run split tests on Cortex and ML models directly in Snowflake, testing performance against actual business metrics.
COMPLETE
method, testing different model versions for various customers. Winning Variant is used to determine which model should be used per customer ID. Later, we would tie business KPIs (i.e., revenue, churn, returns, etc) back to each variant to see which model should be deployed.
We have the following declared for this example:
llama3.2-1b
model) and treatment (claude-3-5-sonnet
)cortex-test
abc123
v1
of our model) and treatment (v2
)ml-test
customer_test_data
in these examples contains the following fields:
customer_id
: ID of the customerage
: how long the customer has been a customerusage
: some arbitrary integer that defines how much the customer has used the productcustomer_churn_model
has been trained to predict churn given an age
and usage
for a customer.
The “Python (Simple)” example below uses the Winning Variant Python SDK.