Optimizations
The top-level entity in Levered. A specific product decision you want to optimize using multi-armed bandits.
An optimization is the core unit of work in Levered. It represents a specific product decision you want to improve: which headline converts best, which checkout flow drives the most revenue, which pricing display maximizes sign-ups.
What an optimization contains
Each optimization bundles everything Levered needs to run and improve a decision:
- Design factors. The variables you control (headline text, button color, layout) and their possible values. See Design Factors.
- A model. The bandit that learns which variants perform best. See Models.
- Linked metrics. The reward signals that define success, written as SQL queries against your warehouse.
- Training data config. How Levered connects exposures to rewards, including the conversion window and warehouse tables.
Example
Suppose you want to optimize a landing page hero section. You might create an optimization with:
- Design factors:
headline(5 options) andcta_text(2 options), producing 10 variants. - Reward metric: a SQL query that counts sign-ups within 24 hours of seeing the page.
- Model: a CMAB that personalizes by country and device type.
Once live, the SDK serves the best variant for each visitor, and the model retrains as new data flows into your warehouse.
Status lifecycle
Every optimization moves through a simple lifecycle:
| Status | Meaning |
|---|---|
draft | Being set up. Nothing is served yet, apart from dry-run calls that verify the SDK integration. |
live | Actively serving variants and collecting data. The model retrains periodically as new observations arrive. |
paused | Temporarily not serving. Users get the fallback defined in your app. Can be resumed. |
shipped | The winner is shipped. Levered serves the leading variant for each context to 100% of traffic. The model no longer learns, there is no holdout, and results are frozen. |
completed | Stopped for good. Levered serves nothing, so users get the fallback defined in your app. Results remain available. |
archived | No longer active or visible in the default dashboard view. Data is retained for reference. |
Ending an optimization: ship or stop
There are two ways to end the learning phase. They differ in what your users get afterwards.
| Ship winner | Stop | |
|---|---|---|
| Status | shipped | completed |
| Users get | The leading variant for their context | The fallback defined in your app |
| Holdout | Removed | Removed |
| Model | Frozen | Frozen |
| Reversible | Yes, resume learning at any time | No |
Shipping serves a policy, not a single variant. If your optimization uses context factors, each context segment the model has seen gets its own leading variant. A context the model has never seen gets the variant that leads over all traffic. The dashboard and levered optimizations show <id> list exactly what is served.
Shipping ends measurement. Lift and guardrails are measured against the holdout. Once the holdout is removed there is nothing left to compare against, so results are frozen as of the moment you ship.
Users switch to the winner. While learning, a returning user keeps the variant they were first assigned. When you ship, everyone gets the winner, including users who were in the holdout.
Your app keeps calling Levered. The SDK integration stays as it is. If you would rather hardcode the winner and remove the integration, read the shipped policy from the dashboard or the CLI, then stop the optimization.
A shipped optimization cannot be archived directly, because it is still serving users. Stop it first.
How this differs from A/B testing
Traditional A/B tests have a fixed design: you choose a sample size, split traffic evenly, wait for statistical significance, then manually pick a winner. This means a lot of traffic goes to underperforming variants while you wait.
Levered optimizations work differently:
- Continuous optimization. The model starts shifting traffic toward better variants as soon as it has enough data. There is no waiting period.
- No fixed sample size. You do not need to calculate a required sample size upfront. The bandit adapts as data accumulates.
- Automatic traffic allocation. Variants that perform well get more traffic. Variants that underperform get less. This happens automatically via Thompson Sampling.
- Exploration never fully stops. Even while exploiting the best variant, the bandit occasionally explores alternatives. If user behavior shifts, the model adapts.
The result: faster decisions, less wasted traffic, and the ability to keep optimizing as conditions change.