Experiments

Experiments enable A/B testing between paywall variants within an audience. Each audience can have one experiment with multiple variants, each mapped to a different paywall.

Create an Experiment

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment

Request Body

Field	Type	Required	Description
`variants`	array	Yes	Array of variant objects (min 1)
`variants[].paywall_id`	string	Yes	Paywall ID for this variant
`variants[].traffic_percentage`	number	Yes	Traffic allocation (0-100)
`confidence_threshold`	number	No	Statistical significance threshold (0-1). Default: `0.95`
`auto_promote`	boolean	No	Auto-promote winner when significance is reached. Default: `false`

If variant traffic percentages sum to less than 100, the remainder becomes the holdout group. Holdout users see no paywall.

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_your_private_key" \
  -d '{
    "variants": [
      { "paywall_id": "pw_original", "traffic_percentage": 50 },
      { "paywall_id": "pw_new_design", "traffic_percentage": 50 }
    ],
    "confidence_threshold": 0.95,
    "auto_promote": true
  }'

Response `201 Created`

{
  "id": "exp_abc123",
  "audienceId": "AUDIENCE_ID",
  "status": "running",
  "holdoutPercentage": 0,
  "confidenceThreshold": 0.95,
  "autoPromote": true,
  "variants": [
    { "id": "var_001", "paywallId": "pw_original", "trafficPercentage": 50 },
    { "id": "var_002", "paywallId": "pw_new_design", "trafficPercentage": 50 }
  ]
}

Get Experiment with Metrics

GET /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment

Returns the experiment with per-variant metrics and statistical significance analysis.

Example

curl https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment \
  -H "Authorization: Bearer sk_your_private_key"

Response `200 OK`

{
  "id": "exp_abc123",
  "audienceId": "AUDIENCE_ID",
  "status": "running",
  "holdoutPercentage": 0,
  "confidenceThreshold": 0.95,
  "autoPromote": true,
  "variants": [
    {
      "id": "var_001",
      "paywallId": "pw_original",
      "trafficPercentage": 50,
      "metrics": {
        "opens": 1250,
        "conversions": 87,
        "conversionRate": 0.0696,
        "revenue": 4350.00
      }
    },
    {
      "id": "var_002",
      "paywallId": "pw_new_design",
      "trafficPercentage": 50,
      "metrics": {
        "opens": 1180,
        "conversions": 112,
        "conversionRate": 0.0949,
        "revenue": 5600.00
      }
    }
  ],
  "significance": {
    "significant": true,
    "confidence": 0.97,
    "winner": "var_002",
    "improvement": 0.364,
    "meetsThreshold": true
  }
}

Significance Fields

Field	Type	Description
`significant`	boolean	Whether the result is statistically significant
`confidence`	number	Confidence level (0-1)
`winner`	string	Variant ID of the likely winner
`improvement`	number	Relative improvement of winner over loser
`meetsThreshold`	boolean	Whether confidence meets the experiment's threshold

Update an Experiment

PUT /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment

Request Body

All fields optional:

Field	Type	Description
`variants`	array	Replace variants (deletes existing, creates new)
`confidence_threshold`	number	New threshold
`auto_promote`	boolean	Enable/disable auto-promote
`status`	string	`running`, `paused`, or `completed`

Promote a Variant

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/promote/:variant_id

Sets the specified variant to 100% traffic and all others to 0%. Marks the experiment as completed.

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/promote/var_002 \
  -H "Authorization: Bearer sk_your_private_key"

Response `200 OK`

{
  "id": "exp_abc123",
  "status": "completed",
  "holdoutPercentage": 0,
  "variants": [
    { "id": "var_001", "paywallId": "pw_original", "trafficPercentage": 0 },
    { "id": "var_002", "paywallId": "pw_new_design", "trafficPercentage": 100 }
  ]
}

Reset an Experiment

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/reset

Resets the experiment status back to running. Does not reset variant assignments or accumulated metrics.

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/reset \
  -H "Authorization: Bearer sk_your_private_key"

Pause an Experiment

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/pause

Pauses a running experiment. While paused, no new users are assigned to variants. Existing variant assignments are preserved.

Only experiments in running status can be paused.

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/pause \
  -H "Authorization: Bearer sk_your_private_key"

Response `200 OK`

{
  "id": "exp_abc123",
  "status": "paused",
  "audienceId": "AUDIENCE_ID",
  "holdoutPercentage": 0,
  "confidenceThreshold": 0.95,
  "autoPromote": true,
  "variants": [
    { "id": "var_001", "paywallId": "pw_original", "trafficPercentage": 50 },
    { "id": "var_002", "paywallId": "pw_new_design", "trafficPercentage": 50 }
  ]
}

Resume an Experiment

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/resume

Resumes a paused experiment, setting its status back to running.

Only experiments in paused status can be resumed.

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/resume \
  -H "Authorization: Bearer sk_your_private_key"

Response `200 OK`

{
  "id": "exp_abc123",
  "status": "running",
  "audienceId": "AUDIENCE_ID",
  "holdoutPercentage": 0,
  "confidenceThreshold": 0.95,
  "autoPromote": true,
  "variants": [
    { "id": "var_001", "paywallId": "pw_original", "trafficPercentage": 50 },
    { "id": "var_002", "paywallId": "pw_new_design", "trafficPercentage": 50 }
  ]
}

Schedule an Experiment

POST /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/schedule

Schedules a draft experiment to start (and optionally end) at specific times.

Only experiments in draft status can be scheduled.

Request Body

Field	Type	Required	Description
`scheduled_start_at`	string	Yes	ISO 8601 datetime for when the experiment should start
`scheduled_end_at`	string	No	ISO 8601 datetime for when the experiment should end

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/schedule \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_your_private_key" \
  -d '{
    "scheduled_start_at": "2025-02-01T00:00:00Z",
    "scheduled_end_at": "2025-02-15T00:00:00Z"
  }'

Response `200 OK`

{
  "id": "exp_abc123",
  "status": "scheduled",
  "scheduledStartAt": "2025-02-01T00:00:00.000Z",
  "scheduledEndAt": "2025-02-15T00:00:00.000Z",
  "audienceId": "AUDIENCE_ID",
  "holdoutPercentage": 0,
  "confidenceThreshold": 0.95,
  "autoPromote": false,
  "variants": [
    { "id": "var_001", "paywallId": "pw_original", "trafficPercentage": 50 },
    { "id": "var_002", "paywallId": "pw_new_design", "trafficPercentage": 50 }
  ]
}

Get Experiment Results

GET /v1/projects/:id/campaigns/:cid/audiences/:aid/experiment/results

Returns comprehensive experiment results including per-variant metrics, statistical significance analysis, sample size estimates, and daily breakdowns.

Example

curl https://agentwallie.com/api/v1/projects/PROJECT_ID/campaigns/CAMPAIGN_ID/audiences/AUDIENCE_ID/experiment/results \
  -H "Authorization: Bearer sk_your_private_key"

Response `200 OK`

{
  "experiment": {
    "id": "exp_abc123",
    "name": "Pricing Test Q1",
    "status": "running",
    "startedAt": "2025-01-15T00:00:00.000Z",
    "durationDays": 12.5
  },
  "variants": [
    {
      "id": "var_001",
      "name": "Control",
      "paywallId": "pw_original",
      "trafficPercentage": 50,
      "isControl": true,
      "metrics": {
        "impressions": 1250,
        "conversions": 87,
        "conversionRate": 0.0696,
        "revenue": 4350.00,
        "revenuePerUser": 3.48,
        "uniqueUsers": 1250,
        "confidenceInterval": { "lower": 0.0564, "upper": 0.0854 }
      }
    },
    {
      "id": "var_002",
      "name": "New Design",
      "paywallId": "pw_new_design",
      "trafficPercentage": 50,
      "isControl": false,
      "metrics": {
        "impressions": 1180,
        "conversions": 112,
        "conversionRate": 0.0949,
        "revenue": 5600.00,
        "revenuePerUser": 4.75,
        "uniqueUsers": 1180,
        "confidenceInterval": { "lower": 0.0791, "upper": 0.1133 }
      }
    }
  ],
  "significance": {
    "method": "frequentist",
    "zScore": 2.31,
    "pValue": 0.0104,
    "confidence": 0.9896,
    "significant": true,
    "winner": "var_002"
  },
  "sampleSize": {
    "currentPerVariant": [1250, 1180],
    "recommendedMinimum": 3200,
    "estimatedDaysRemaining": 19.8
  },
  "daily": [
    { "date": "2025-01-15", "variantId": "var_001", "impressions": 95, "conversions": 6, "revenue": 299.94 },
    { "date": "2025-01-15", "variantId": "var_002", "impressions": 88, "conversions": 9, "revenue": 449.91 }
  ]
}

Results Fields

`significance` (Frequentist)

Field	Type	Description
`method`	string	`"frequentist"`
`zScore`	number	Z-score of the comparison
`pValue`	number	p-value (lower = more significant)
`confidence`	number	Confidence level (0-1)
`significant`	boolean	Whether the result meets the confidence threshold
`winner`	string	Variant ID of the likely winner

`significance` (Bayesian)

Field	Type	Description
`method`	string	`"bayesian"`
`probabilityBest`	object	Map of variant ID to probability of being the best (e.g., `{"var_001": 0.12, "var_002": 0.88}`)
`expectedLoss`	object	Map of variant ID to expected loss if chosen as winner
`winner`	string	Variant ID with highest probability of being best

`sampleSize`

Field	Type	Description
`currentPerVariant`	number[]	Current impressions per variant
`recommendedMinimum`	number \| null	Recommended minimum sample size per variant
`estimatedDaysRemaining`	number \| null	Estimated days until recommended sample size is reached

Estimate Sample Size

POST /v1/projects/:id/experiments/estimate-sample-size

A utility endpoint to calculate the recommended sample size per variant before creating an experiment.

This endpoint is at the project level, not nested under an audience or experiment.

Request Body

Field	Type	Required	Description
`baseline_rate`	number	Yes	Expected baseline conversion rate (0.0001-0.9999)
`minimum_detectable_effect`	number	Yes	Smallest relative change you want to detect (e.g., `0.1` for 10% improvement)
`alpha`	number	No	Significance level. Default: `0.05`
`power`	number	No	Statistical power. Default: `0.80`

Example

curl -X POST https://agentwallie.com/api/v1/projects/PROJECT_ID/experiments/estimate-sample-size \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_your_private_key" \
  -d '{
    "baseline_rate": 0.05,
    "minimum_detectable_effect": 0.2,
    "alpha": 0.05,
    "power": 0.80
  }'

Response `200 OK`

{
  "sample_size_per_variant": 3842,
  "baseline_rate": 0.05,
  "minimum_detectable_effect": 0.2,
  "alpha": 0.05,
  "power": 0.80
}

Use this before creating an experiment to understand how much traffic you need. For example, with a 5% baseline conversion rate and a goal of detecting a 20% relative improvement, you would need approximately 3,842 impressions per variant.

Experiment Status Lifecycle

Experiments follow a defined status lifecycle:

draft -> scheduled -> running -> paused -> running -> completed
         |                                             ^
         +-> running ----> completed                   |
                    +----> paused -> running -----------+

Transition	Action	Endpoint
`draft` -> `running`	Start	`POST .../experiment/start`
`draft` -> `scheduled`	Schedule	`POST .../experiment/schedule`
`scheduled` -> `running`	Start	`POST .../experiment/start`
`running` -> `paused`	Pause	`POST .../experiment/pause`
`paused` -> `running`	Resume	`POST .../experiment/resume`
`running` -> `completed`	Complete	`POST .../experiment/complete`
`running` -> `completed`	Promote	`POST .../experiment/promote/:variant_id`
Any -> `running`	Reset	`POST .../experiment/reset`

⚠️

Experiments can only be deleted when in draft or completed status. Pause or complete a running experiment before deleting.

Auto-Promote Behavior

When auto_promote is enabled on an experiment:

After each batch of events is ingested, the system checks experiment metrics.
If one variant's conversion rate is significantly better than the other(s) and the confidence meets the threshold, the winner is automatically promoted.
The experiment status changes to completed and the winning variant gets 100% traffic.

This runs asynchronously during event ingestion and does not block the event response.

Audiences Users

Experiments

Create an Experiment

Request Body

Example

Response 201 Created

Get Experiment with Metrics

Example

Response 200 OK

Significance Fields

Update an Experiment

Request Body

Promote a Variant

Example

Response 200 OK

Reset an Experiment

Example

Pause an Experiment

Example

Response 200 OK

Resume an Experiment

Example

Response 200 OK

Schedule an Experiment

Request Body

Example

Response 200 OK

Get Experiment Results

Example

Response 200 OK

Results Fields

significance (Frequentist)

significance (Bayesian)

sampleSize

Estimate Sample Size

Request Body

Example

Response 200 OK

Experiment Status Lifecycle

Auto-Promote Behavior

Response `201 Created`

Response `200 OK`

Response `200 OK`

Response `200 OK`

Response `200 OK`

Response `200 OK`

Response `200 OK`

`significance` (Frequentist)

`significance` (Bayesian)

`sampleSize`

Response `200 OK`