Inhomogeneous Bernoulli Process

Reference

Application: conversion rate optimization when all groups share a common multiplicative time-varying effect.

Suppose a new experimental unit $n$ is randomly assigned to one of $i\in \{1,2,3\}$ experiment treatment groups at time $t$, with assignment probabilities $\rho =[0.1,0.3,0.6]$, and a Bernoulli outcome is observed with probability $p_i(t)=\exp (\mu (t)+{\delta }_i)$, $\delta =[\log 0.2,\log 0.3,\log 0.4]$. The conditional probability that the next Bernoulli success comes from group $i$ is

$${\theta }_i=\frac{{\rho }_i\exp ({\delta }_i)}{\sum _{j=1}^d{\rho }_j\exp ({\delta }_j)}.$$

Therefore, the next Bernoulli success comes from a random group, $\mathrm{Multinomial}(1,\theta )$ distributed, with $\theta \approx [0.05,0.25,0.68]$.

import numpy as np

rho = np.array([0.1, 0.3, 0.6])
delta = np.log([0.2, 0.3, 0.4])
theta = rho * np.exp(delta) / np.sum(rho * np.exp(delta))
size = 4000
np.random.seed(1)
xs = np.random.multinomial(1, theta, size=size)
print(xs)

[[0 0 1]
 [0 0 1]
 [0 0 1]
 ...
 [0 1 0]
 [0 1 0]
 [0 0 1]]

We can test the hypothesis

$$\begin{array}{r}{H}_0:{\delta }_0\ge {\delta }_1,{\delta }_0\ge {\delta }_2\\ H_1:{\delta }_0<{\delta }_1,{\delta }_0<{\delta }_2\end{array}$$

using a Multinomial test with ${\theta }_0=\rho$ and a list of inequalities for $\delta$. To estimate $(1-\alpha )$ confidence intervals for the contrasts, we may set a matrix of weights, with rows $[-1,0,1]$ for ${\delta }_2-{\delta }_0$ and $[0,-1,1]$ for ${\delta }_2-{\delta }_1$:

from savvi.multinomial import InhomogeneousBernoulliProcess

alpha = 0.05
hypothesis = lambda delta: [delta[0] >= delta[1], delta[0] >= delta[2]]
weights = np.array([[-1, 0, 1], [0, -1, 1]])
ibp = InhomogeneousBernoulliProcess(alpha, rho, hypothesis, weights)

AttributeError: _ARRAY_API not found

(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver ECOS:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver SCS:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver ECOS_BB:
ImportError('numpy.core.multiarray failed to import')
(CVXPY) Oct 24 05:37:32 PM: Encountered unexpected exception importing solver OSQP:
ImportError('numpy.core.multiarray failed to import')

AttributeError: _ARRAY_API not found

AttributeError: _ARRAY_API not found

AttributeError: _ARRAY_API not found

For each new unit sample $n$, we run the test. If $p_n<\alpha$, we have the option to stop running:

sequence = ibp.batch(xs)
optional_stop = next(s for s in sequence if s.p_value <= alpha)
optional_stop

Parameter	Estimate	CI Lower	CI Upper
$-{\delta }_0+{\delta }_2$	0.8232	0.0009	1.7716
$-{\delta }_1+{\delta }_2$	0.2231	-0.1577	0.6472

Sample size: 302, P-value: 0.0492

%config InlineBackend.figure_formats = ["svg"]
import matplotlib.pyplot as plt
from savvi.utils import plot

contrasts = ibp.weights @ delta
_, ax1, _ = plot(sequence, contrasts)
ax1.set_ylim(-1, 2)
plt.show()