# Run a holdout

> Start a sticky 50/50 test on a surface and know the day it becomes readable.

Source: https://www.patterna.nl/docs/run-a-holdout

A holdout splits users into two arms on one surface. Treatment gets Patterna's slate. Control gets the surface's editorial list. The ledger reports the difference per served slate. Nothing else counts as proof.

## Start one in the panel

1. Open **Proof → Experiments** and choose **Start a holdout**.
2. Pick the surface. It should be `live`.
3. Fill in the fields below. The panel computes the readable sample and the projected date as you type, with a power curve.
4. Start. The experiment appears in the list with users per arm.

| Field | What it means | Default |
|---|---|---|
| `surface` | The surface key, e.g. `home_articles` | required |
| `name` | A name you will recognise in the history | required |
| `primary_metric` | `tap_rate`, `second_item_rate`, `join_rate` (community surfaces only), `engaged_after_tap` or `purchase_rate` (product surfaces only), defined in [Proof and the ledger](https://www.patterna.nl/docs/proof-and-the-ledger) | required |
| `treatment_share` | Share of units in treatment, above 0 and at most 1 | `0.5` |
| `stratify_by` | `none` (default): the split is a plain hash of the unit. `persona` draws the arm within each persona stratum; users without a persona, including `non_profiled` traffic, form their own stratum | `none` |
| `baseline_rate` | Today's rate of the metric, e.g. `0.042`: your own analytics for the block you replace, or, for a new block, the surface's control rate after a week in `testers` or `live` (Proof → the surface shows it). Without it the row uses the observed control rate and a 20% change | none |
| `relative_change` | The change you want to be able to detect, e.g. `0.2` for +20% | none |

Baseline and change to detect set the **readable sample**: how many units per arm the test needs before the interval means something. With today's traffic Patterna projects **`readable_at`**, the date that sample is reached. A small change to detect needs a large sample. If the date is too far away, test for a larger change or pick a busier surface. A ledger row is `readable` only when each arm has reached `readable_sample`.

One experiment per surface runs at a time. Starting a second answers `409 experiment_running`.

## Start one through the API

```curl
curl -sS -X POST "https://patterna-api.orven.am/v1/experiments" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "surface": "home_articles",
    "name": "Home articles pilot holdout",
    "primary_metric": "tap_rate",
    "treatment_share": 0.5,
    "baseline_rate": 0.042,
    "relative_change": 0.2
  }'
```

The response is the experiment with `id`, `started_at`, `readable_sample` and `readable_at`. List all experiments with `GET https://patterna-api.orven.am/v1/experiments`. `join_rate` on a surface without community items is refused with `400 metric_not_applicable`.

## How units are assigned

- **Sticky.** The arm comes from a hash of the experiment id and the user id, and is stored, so a user stays in the same arm for the whole experiment.
- **Sessions in `non_profiled` mode.** The arm comes from a hash of the session id, so opted-out users are covered too. When your app rotates session ids, the same person can land in both arms over a day, and the interval is clustered on sessions, not people. To prevent that, send `assignment_unit`: your backend sends an HMAC-SHA256 of its own user id (64 hex characters), Patterna uses it only to pick the arm and the `unit_hash`, and never stores it raw. See [Identity across connectors](https://www.patterna.nl/docs/identity-across-connectors).
- **Same UI in both arms.** Only the order and choice of items differ. Do not give one arm a different card or label.

## What the control arm sees

Control gets the surface's editorial list: newest items, a curated list of ids, or the 7-day popular list. It is logged with policy `editorial`, propensity 1 and a reason for every slot. It is not a fallback and does not count in the fallback rate.

On an anchored surface such as "more to read" under an article, one list for everyone would compare anchored ranking against a list unrelated to the page, so send your own list for this page in `context.control_item_ids` (for example the article's pairs-with products, or the editor's related articles), or set the surface's `fallback.by_anchor` (an object from anchor item id to a list of ids). In order, control serves `context.control_item_ids`, else `fallback.by_anchor` for the anchor, else the surface's own list, with the anchor, `exclude_item_ids`, joined communities, the candidate kinds, premium and locale applied, and newest items filling the rest. The answer's `control_source` says which. That is the comparison you want: Patterna against what you would show without it.

When treatment ends up shorter than requested after its filters, it is filled from the same editorial list, and the slate says how many items with `fill_count` (each filled slot is logged with `fill: true`, propensity 1).

## Do not re-sort on the device

A surface under an experiment must not be re-sorted by the app. The ledger and the off-policy estimator read the position Patterna served. If the app moves items around, the arms are no longer compared on what was served. The exception is a surface with `device_resort: true`, as long as every interaction carries `displayed_position` (else `422 displayed_position_required`).

## Report honestly

- Report seen and taps for both arms the same way. A bug that loses one arm's events shows up as SRM, or worse, as a fake lift.
- When your backend gives up waiting and shows its own list, or hides a block it cannot fill, the slate still counts in its arm with no taps. The SDKs report this as abandonment; do the same from your own client.
- When the answer says `shown: false` (fewer items than the surface's `min_slots`), it carries no items: hide the block and report nothing, not even an abandonment (abandon answers `409 not_shown`). Such slates count in their arm as served with zero success.
- When a user did not allow usage analytics and you will not report anything, send `context.measured: false` so those slates stay out of the ledger in both arms.

See [Timeouts and what counts in the ledger](https://www.patterna.nl/docs/timeouts-and-the-ledger).

## Watch for SRM

A sample-ratio mismatch (SRM) means the arms are not the size you asked for. For a 50/50 test, 5,000 against 4,000 users is a red flag. It usually points to a bug: one arm's events get lost, or the app fails to render one arm. Patterna checks every ledger row. A row with an SRM p-value below 0.001 is flagged and should not be read. Fix the cause before you trust the numbers.

## End a holdout

End it from **Proof → Experiments**, or:

```curl
curl -sS -X POST "https://patterna-api.orven.am/v1/experiments/$EXPERIMENT_ID/end" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY"
```

The experiment keeps its history and its ledger rows; the ledger window closes at `ended_at`. From then on the surface's slates are logged with arm `none`. A common choice after a clear result is a new experiment with a permanent 10% control (`treatment_share: 0.9`), so the ledger keeps counting.

## Next

- [Read a ledger row](https://www.patterna.nl/docs/read-a-ledger-row)
- [Proof and the ledger](https://www.patterna.nl/docs/proof-and-the-ledger)