Exploration and propensities

On a share of requests Patterna does not show its top list in fixed order. It samples the order instead and logs, for every slot, the exact probability that the item landed there. That probability is the propensity. With propensities in the log you can estimate how a different flow would have done on the same traffic, without shipping it.

Why explore at all#

If a surface always shows the same top six, the log only tells you how those six do in those positions. A new flow that would put item 9 first cannot be evaluated: item 9 was never shown first. Exploration gives every eligible item a known, non-zero chance at each position. That overlap is what makes offline evaluation possible.

The settings#

Exploration is part of the flow config:

{
  "exploration": { "kind": "plackett_luce", "temperature": 0.7, "share": 0.2 }
}

When the flow's share is 0, the surface's exploration_share applies. A share above 0.5 is refused when you save the flow.

Which slates are explored#

Exploration runs on slates in the treatment arm of a running holdout and on slates outside any holdout (arm none: no experiment, shadow, testers). A control slate is never explored: it gets the editorial list. So a surface collects propensity-logged data before its first holdout.

What learns from those slates: the slate learners (the persona model, the learned linear ranker, the gbdt ranker and the SNIPS gate) train on slates in arm treatment and arm none, never on control, and only on measured, shown, non-shadow slates. Their feedback is what you report with POST /v1/interactions (seen, taps, dismissals), joined to the slate by request_id, plus signed item_completed events for an item served under that request (counted as a tap). Positions filled from the editorial list (fill: true) are never used as ranking evidence.

Plackett-Luce sampling#

On an explored request the ranker's scores turn into weights. Patterna fills the slate one position at a time:

1. Draw one item from the eligible pool, with probability proportional to its weight.
2. Record that probability as the propensity for this position.
3. Remove the item from the pool and repeat for the next position.

Each logged propensity is exact: it is the probability over the pool the item was actually drawn from. The engine's tests check that empirical frequencies match the stated probabilities. On requests that are not explored, the slate is the deterministic top-k.

Why caps live inside the sampler#

Flows can cap items per collection (per_collection_cap, 0 = no cap) and premium items (premium_policy: cap with premium_cap, default 2). There are two ways to enforce a cap:

• After sampling: draw six, then drop the fourth recipe and pull up the next item. The item now at position 4 was never drawn at position 4. Its logged propensity is wrong, and every estimate built on it is biased.
• Inside the sampler: before each draw, remove items that would break a cap from the pool. The propensity is the probability over the pool that remained. It stays exact.

Patterna does the second. Diversity works the same way: filters.diversity (0 to 1, 0 = off) lowers an item's score at each position by that factor times its highest tag similarity to the items already picked, before the draw. Nothing re-ranks after sampling. PropensityAt in the engine recomputes the same number for any logged slate, which is what the simulate endpoint uses.

filters.diversity is the only diversity setting. An old top-level diversity object is dropped on save.

SNIPS: estimating a flow on logged traffic#

Self-normalised inverse propensity scoring (SNIPS) answers: if the candidate flow had served these requests, what would the metric have been?

For each logged slot i with outcome r_i (a tap, say), weight it by how much more or less likely the candidate flow is to put that item there than the logging flow was:

w_i     = p_candidate(item at position | request) / p_logged(item at position | request)
SNIPS   = Σ w_i · r_i / Σ w_i
ESS     = (Σ w_i)² / Σ w_i²

The effective sample size (ESS) tells you how much traffic the estimate really rests on. If the candidate flow differs a lot from what was logged, a few slots get huge weights, ESS collapses, and the interval widens. Treat a small ESS as "not enough evidence", not as a result.

Run it from Flows → Simulate, or:

curl -sS -X POST "https://patterna-api.orven.am/v1/surfaces/home_articles/simulate" \
  -H "Authorization: Bearer $PATTERNA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "days": 14,
    "config": {
      "generators": [
        { "kind": "content_affinity", "weight": 0.6 },
        { "kind": "trending", "weight": 0.4 }
      ],
      "ranker": { "kind": "linear" },
      "exploration": { "kind": "plackett_luce", "temperature": 0.7, "share": 0.2 }
    }
  }'

The answer holds current_policy, candidate_policy, ci_low, ci_high, impressions_used and effective_sample_size, plus the twin curve when the tenant has one. Nothing changes on the surface.

SNIPS is an estimate. It tells you which flows are worth a holdout. Only the holdout is proof. See Proof and the ledger.

Positions must be the served ones#

Propensities belong to the position Patterna served, counted from 0. Always report the served position. A surface under a holdout must not be re-sorted on the device unless it has device_resort: true: the ledger and the estimator read the served position.

With device_resort: true on a surface the app may re-sort with its own rule and must send displayed_position on every interaction (else 422 displayed_position_required); the estimators use it to model where the user actually looked. See Surfaces and flows.

Read next#

• The twin
• Flow config reference
• Surfaces and flows