Inhomogeneous Bernoulli Process

Application: conversion rate optimization when all groups share a common multiplicative time-varying effect.

Suppose a new experimental unit \(n\) is randomly assigned to one of \(i \in \{1, 2, 3\}\) experiment treatment groups at time \(t\), with assignment probabilities \(\mathbf{\rho} = [0.1, 0.3, 0.6]\), and a Bernoulli outcome is observed with probability \(p_i(t) = \exp(\mu(t) + \delta_{i})\), \(\mathbf{\delta} = [\log 0.2, \log 0.3, \log 0.4]\). The conditional probability that the next Bernoulli success comes from group \(i\) is

\[ \theta_i = \frac{\rho_i \exp(\delta_i)}{\sum_{j=1}^d \rho_j \exp(\delta_j)}. \]

Therefore, the next Bernoulli success comes from a random group, \(\mathrm{Multinomial}(1, \mathbf{\theta})\) distributed, with \(\mathbf{\theta} \approx [0.05, 0.25, 0.68]\).

import numpy as np

rho = np.array([0.1, 0.3, 0.6])
delta = np.log([0.2, 0.3, 0.4])
theta = rho * np.exp(delta) / np.sum(rho * np.exp(delta))
size = 4000
np.random.seed(1)
xs = np.random.multinomial(1, theta, size=size)
print(xs)

[[0 0 1]
 [0 0 1]
 [0 0 1]
 ...
 [0 1 0]
 [0 1 0]
 [0 0 1]]

We can test the hypothesis

\[ \begin{align} H_0: \delta_0 \geq \delta_1, \delta_0 \geq \delta_2 \\ H_1: \delta_0 \lt \delta_1, \delta_0 \lt \delta_2 \end{align} \]

using a Multinomial test with \(\mathbf{\theta}_0 = \mathbf{\rho}\) and a list of inequalities for \(\mathbf{\delta}\). To estimate \((1 - \alpha)\) confidence intervals for the contrasts, we may set a matrix of weights, with rows \([-1, 0, 1]\) for \(\delta_2 - \delta_0\) and \([0, -1, 1]\) for \(\delta_2 - \delta_1\):

from savvi.multinomial import InhomogeneousBernoulliProcess

alpha = 0.05
hypothesis = lambda delta: [delta[0] >= delta[1], delta[0] >= delta[2]]
weights = np.array([[-1, 0, 1], [0, -1, 1]])
ibp = InhomogeneousBernoulliProcess(alpha, rho, hypothesis, weights)

AttributeError: _ARRAY_API not found

(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver ECOS:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver SCS:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver ECOS_BB:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver OSQP:
ImportError('numpy.core.multiarray failed to import')

AttributeError: _ARRAY_API not found

AttributeError: _ARRAY_API not found

AttributeError: _ARRAY_API not found

For each new unit sample \(n\), we run the test. If \(p_n < \alpha\), we have the option to stop running:

sequence = ibp.batch(xs)
optional_stop = next(s for s in sequence if s.p_value <= alpha)
optional_stop

Parameter	Estimate	CI Lower	CI Upper
\(- \delta_0 + \delta_2\)	0.8232	0.0009	1.7716
\(- \delta_1 + \delta_2\)	0.2231	-0.1577	0.6472

Sample size: 302, P-value: 0.0492

%config InlineBackend.figure_formats = ["svg"]
import matplotlib.pyplot as plt
from savvi.utils import plot

contrasts = ibp.weights @ delta
_, ax1, _ = plot(sequence, contrasts)
ax1.set_ylim(-1, 2)
plt.show()