Proof and the ledger

Proof in Patterna means one thing: on a surface, users who get recommendations are compared with users who get your editorial list, and the difference has an interval. Dashboards of clicks, revenue attributed by last touch, or offline scores are useful. They are not proof.

The holdout#

A surface gets a holdout when you start an experiment on it. While it runs there are two arms.

• Treatment gets Patterna's ranked slate.
• Control gets the surface's editorial list: the newest items, a curated list of ids, or the 7-day popular list, logged with policy editorial. It is the comparison, not a fallback, and it never counts in the fallback rate.

On an anchored surface (a page about one item, such as "more to read" under an article) one list for everyone would compare Patterna with a list that has nothing to do with the page, so control is anchored. In order, control serves the ids you send in context.control_item_ids (your own list for this page, for example the article's pairs-with products or the editor's related articles), else the surface's list for this anchor from fallback.by_anchor, else the surface's own list. The engine applies the same rules as in treatment (the anchor is never served; exclude_item_ids, joined communities, candidate kinds, premium and locale apply) and fills from newest items when the list is short. The answer's control_source (request, by_anchor or surface) says which list control served. The holdout then compares Patterna with what your editor would have shown on that page. Do not open a holdout on an anchored surface before you send control_item_ids or have set fallback.by_anchor.

Without a running experiment every slate is logged with arm none. It counts in the surface metrics, never in the ledger.

Rules:

• Sticky. A user stays in one arm for the whole experiment: the arm comes from a hash of the experiment and the user id, and is stored. In non_profiled mode it comes from a hash of the session id, so a new session can land in the other arm, unless you send assignment_unit on non_profiled requests: 64 hex characters, an HMAC-SHA256 your backend computes from its own user id, so a person keeps one arm across sessions without Patterna storing who she is. See Identity across connectors.
• Stratification. Without stratify_by (default none) the arm is a plain hash of the unit. With stratify_by: "persona" the arm is drawn within each persona, so both arms get the same mix of personas; strata_shares can set the treatment share per persona stratum. Units without a persona (including non_profiled traffic) form their own stratum.
• Same UI. Both arms see the same placement and the same card design. Only the ordering differs.
• Both arms are logged. Control slates are logged with propensity 1 and a reason, so "every slate is logged" and "a reason for every slot" hold in both arms.
• No re-sorting. A surface under an experiment must not be re-sorted on the device, unless the surface has device_resort: true.

During a pilot the split is usually 50/50.

What counts: intention to treat#

Every served slate counts in its arm, seen or not. A slate the app gave up on stays in its arm with no taps; dropping it would flatter whichever arm is slower. A fallback slate (timeout, storage trouble) is logged under the arm the unit was assigned to, with fallback: true. The ledger reads only impressions inside the experiment's window, from started_at to ended_at, intersected with the period you ask for.

One kind of slate is left out: slates requested with context.measured: false (the user did not allow usage analytics, so nothing will be reported). Consent decides that before the arm is drawn, so leaving these slates out of both arms keeps the comparison fair.

Slates answered with shown: false (fewer items than the surface's min_slots passed the request's rules) also count in their arm as served with zero success, like a slate the app gave up on. They are answered with no items. min_slots is counted before the arm, so both arms hide alike. (Engine 0.2 left these slates out, and since control is padded to full length and treatment could lose items to its filters, that exclusion biased the ledger.) A treatment slate that falls short is filled from the editorial list, like control, and reports how many items it filled in fill_count. Details and a table of every case: Timeouts and what counts in the ledger.

What is measured#

When interactions carry a block (one slate shown in two places on a page), the ledger can also give a row per block (by=block), so the top of a page and its end can be looked at separately. Those rows are descriptive: the whole-slate row decides.

Retention and conversion are reported, not proven, until the user base is large enough to detect a change in them. Revenue is not computed yet; when it is, it will be a last-touch diagnostic line, never the result.

The ledger#

The ledger is the record per surface. Each row holds the control rate, the treatment rate, their difference (treatment minus control), a 95% interval, the slates and the units per arm, the SRM p-value and a flag that says whether the row is readable. Synthetic rows are excluded. Every impression also records the model versions that ranked it (model_versions), visible in the trace and the export. The jobs write a snapshot for the previous month on the 1st at 04:00 UTC; the panel also shows weekly rows per arm. Walkthrough: Read a ledger row.

Why slates are not units#

The metric is per slate, but the randomised unit is the user (or the session in non_profiled mode). One user produces many slates, and those are not independent: a user who taps a lot taps in both her first and her tenth slate. If you treat every slate as a separate observation, you pretend to have far more evidence than you do, and the interval comes out too narrow.

So Patterna clusters the interval on the randomised unit. Per arm, with y the tapped slates and x the served slates of each unit, the rate is R = Σy / Σx. Its variance comes from the delta method over units:

Var(R) ≈ ( s_y² − 2·R·s_xy + R²·s_x² ) / ( n · x̄² )

with n units, x̄ the mean slates per unit, and s the sample variances and covariance across units. The interval of the difference is difference ± 1.96 · √(Var_treatment + Var_control).

The unit is stored with each impression as unit_hash, a SHA-256 of a secret salt Patterna holds per tenant and the user id. It survives a deletion request, so unit counts and intervals do not change after the fact. The salt is set once per tenant and never rotated: rotating it would split every unit, so it is never changed inside an experiment. It cannot be reversed, and someone who holds a user id cannot check whether a row belongs to it.

Sample ratio mismatch (SRM)#

If you asked for 50/50 and got 5,000 users in one arm and 4,600 in the other, something is broken: assignment, logging, or a filter that drops one arm more often. The ledger tests the unit counts per arm against the planned share. A row with an SRM p-value below 0.001 is flagged with a note and should not be read, however good the difference looks. Fix the cause first. The flag is on the row; SRM alerts to the notification centre, email or Slack are not written yet.

Readable#

When you start an experiment with a baseline rate and the change you want to detect, Patterna computes the units per arm needed to see that change at a two-sided 5% level with 80% power (readable_sample) and projects readable_at from current traffic.

A row is flagged readable when each arm has reached the experiment's readable_sample and the interval exists. Do not stop an experiment early because a dashed number looks good.

Check it yourself#

Export a period from Proof → Export, or call the API:

curl -sS -X POST "https://patterna-api.orven.am/v1/proof/export" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"from": "2026-09-01", "to": "2026-09-30", "surface": "home_articles"}'

The export holds impressions and interactions for the period, so you can check the slate and tap counts. It does not yet carry the clustering unit (unit_hash), measured, not_shown, tester, fallback, blocks or the experiment window, so it cannot reproduce a ledger row with its interval on its own; that is an open item on our side. Exports are written by the engine to its export directory; an S3 target is not built yet.

Before going live: the replay report#

Proof starts after launch, but you can get an estimate before. Given 90 days of events and the catalog, the replay report shows what the first flow would have done against the current ordering, with a SNIPS estimate and its effective sample size. It is an estimate, not proof. See Exploration and propensities.

Read next#

• Timeouts and what counts in the ledger
• Run a holdout
• Read a ledger row