Run a holdout

A holdout splits users into two arms on one surface. Treatment gets Patterna's slate. Control gets the surface's editorial list. The ledger reports the difference per served slate. Nothing else counts as proof.

Start one in the panel#

1. Open Proof → Experiments and choose Start a holdout.
2. Pick the surface. It should be live.
3. Fill in the fields below. The panel computes the readable sample and the projected date as you type, with a power curve.
4. Start. The experiment appears in the list with users per arm.

Baseline and change to detect set the readable sample: how many units per arm the test needs before the interval means something. With today's traffic Patterna projects readable_at, the date that sample is reached. A small change to detect needs a large sample. If the date is too far away, test for a larger change or pick a busier surface. A ledger row is readable only when each arm has reached readable_sample.

One experiment per surface runs at a time. Starting a second answers 409 experiment_running.

Start one through the API#

curl -sS -X POST "https://patterna-api.orven.am/v1/experiments" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "surface": "home_articles",
    "name": "Home articles pilot holdout",
    "primary_metric": "tap_rate",
    "treatment_share": 0.5,
    "baseline_rate": 0.042,
    "relative_change": 0.2
  }'

The response is the experiment with id, started_at, readable_sample and readable_at. List all experiments with GET https://patterna-api.orven.am/v1/experiments. join_rate on a surface without community items is refused with 400 metric_not_applicable.

How units are assigned#

• Sticky. The arm comes from a hash of the experiment id and the user id, and is stored, so a user stays in the same arm for the whole experiment.
• Sessions in non_profiled mode. The arm comes from a hash of the session id, so opted-out users are covered too. When your app rotates session ids, the same person can land in both arms over a day, and the interval is clustered on sessions, not people. To prevent that, send assignment_unit: your backend sends an HMAC-SHA256 of its own user id (64 hex characters), Patterna uses it only to pick the arm and the unit_hash, and never stores it raw. See Identity across connectors.
• Same UI in both arms. Only the order and choice of items differ. Do not give one arm a different card or label.

What the control arm sees#

Control gets the surface's editorial list: newest items, a curated list of ids, or the 7-day popular list. It is logged with policy editorial, propensity 1 and a reason for every slot. It is not a fallback and does not count in the fallback rate.

On an anchored surface such as "more to read" under an article, one list for everyone would compare anchored ranking against a list unrelated to the page, so send your own list for this page in context.control_item_ids (for example the article's pairs-with products, or the editor's related articles), or set the surface's fallback.by_anchor (an object from anchor item id to a list of ids). In order, control serves context.control_item_ids, else fallback.by_anchor for the anchor, else the surface's own list, with the anchor, exclude_item_ids, joined communities, the candidate kinds, premium and locale applied, and newest items filling the rest. The answer's control_source says which. That is the comparison you want: Patterna against what you would show without it.

When treatment ends up shorter than requested after its filters, it is filled from the same editorial list, and the slate says how many items with fill_count (each filled slot is logged with fill: true, propensity 1).

Do not re-sort on the device#

A surface under an experiment must not be re-sorted by the app. The ledger and the off-policy estimator read the position Patterna served. If the app moves items around, the arms are no longer compared on what was served. The exception is a surface with device_resort: true, as long as every interaction carries displayed_position (else 422 displayed_position_required).

Report honestly#

• Report seen and taps for both arms the same way. A bug that loses one arm's events shows up as SRM, or worse, as a fake lift.
• When your backend gives up waiting and shows its own list, or hides a block it cannot fill, the slate still counts in its arm with no taps. The SDKs report this as abandonment; do the same from your own client.
• When the answer says shown: false (fewer items than the surface's min_slots), it carries no items: hide the block and report nothing, not even an abandonment (abandon answers 409 not_shown). Such slates count in their arm as served with zero success.
• When a user did not allow usage analytics and you will not report anything, send context.measured: false so those slates stay out of the ledger in both arms.

See Timeouts and what counts in the ledger.

Watch for SRM#

A sample-ratio mismatch (SRM) means the arms are not the size you asked for. For a 50/50 test, 5,000 against 4,000 users is a red flag. It usually points to a bug: one arm's events get lost, or the app fails to render one arm. Patterna checks every ledger row. A row with an SRM p-value below 0.001 is flagged and should not be read. Fix the cause before you trust the numbers.

End a holdout#

End it from Proof → Experiments, or:

curl -sS -X POST "https://patterna-api.orven.am/v1/experiments/$EXPERIMENT_ID/end" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY"

The experiment keeps its history and its ledger rows; the ledger window closes at ended_at. From then on the surface's slates are logged with arm none. A common choice after a clear result is a new experiment with a permanent 10% control (treatment_share: 0.9), so the ledger keeps counting.

Next#

• Read a ledger row
• Proof and the ledger