File:Illustration of Bayesian AB test analysis.svg

Summary

Description
English: Top: a Normal probability distribution with mean 0 and standard deviation 0.3, encoding belief that 68% of experiments have a lift between -30% and 30% (see 68–95–99.7 rule). Bottom: posterior distribution (also a Normal, due to conjugacy), showing how beliefs about likely values of lift shifted after collecting data. The area to the right of 0 is highlighted, indicating the chance to win (probability that lift is greater than 0).
Date
Source Own work
Author Mikhail Popov
SVG development
InfoField
 The source code of this SVG is invalid due to 6 errors.
 This W3C-invalid chart was created with R.
Category:Invalid SVG created with R code#0167Illustration%20of%20Bayesian%20AB%20test%20analysis.svg
Source code
InfoField

R code

library(tidyverse)
library(ggdist)
library(distributional)
library(patchwork) # install.packages("patchwork")

# implementation of https://docs.growthbook.io/statistics/details
calculate_posterior <- function(metric_stats, mu_rel_prior = 0, s2_rel_prior = 0.09) {
    # extract sample sizes:
    n_C <- metric_stats['control', 'sample_size']
    n_T <- metric_stats['treatment', 'sample_size']

    # extract sample means:
    mu_C <- metric_stats['control', 'sample_mean']
    mu_T <- metric_stats['treatment', 'sample_mean']

    # extract sample variances:
    s2_C <- metric_stats['control', 'sample_variance']
    s2_T <- metric_stats['treatment', 'sample_variance']

    # calculate lift (relative effect)
    delta_rel <- (mu_T - mu_C)/mu_C
    s2_delta_rel <- ((s2_C * mu_T**2)/(mu_C**4 * n_C)) + ((s2_T)/(mu_C**2 * n_T))

    # mean and variance for the posterior of relative effect:
    mu_rel_posterior <- (
        # Numerator:
        ((mu_rel_prior/s2_rel_prior) + (delta_rel/s2_delta_rel)) /
            # Denominator:
            ((1/s2_rel_prior) + (1/s2_delta_rel))
    )
    s2_rel_posterior <- 1 / ((1/s2_rel_prior) + (1/s2_delta_rel))

    delta_rel_posterior <- dist_normal(mu_rel_posterior, sqrt(s2_rel_posterior))

    return(list(
        quantities = list(
            sample_sizes = c(control = n_C, treatment = n_T),
            sample_means = c(control = mu_C, treatment = mu_T),
            sample_variances = c(control = s2_C, s2_T),
            lift = c(estimate = delta_rel, variance = s2_delta_rel)
        ),
        distribution = delta_rel_posterior
    ))
}

example_stats <- matrix(
    c(
        # control:
        1e3, # sample size
        0.13, # sample mean
        0.13 * (1 - 0.13), # sample variance
        # treatment:
        1e3, # sample size
        0.145, # sample mean (10% lift from control)
        0.145 * (1 - 0.145) # sample variance
    ),
    nrow = 2, ncol = 3, byrow = TRUE,
    dimnames = list(
        c("control", "treatment"),
        c("sample_size", "sample_mean", "sample_variance")
    )
)

example_posterior <- calculate_posterior(example_stats)

theme_set(theme_ggdist(base_family = "Arial"))

# prior
p1 <- tibble(dist = dist_normal(0, 0.3))

Licensing

I, the copyright holder of this work, hereby publish it under the following license:
Category:Self-published work
Copyright in this work is either owned/co-owned by the Wikimedia Foundation or the content has been licensed to the Wikimedia Foundation. The uploader asserts that they are acting as an agent for the Wikimedia Foundation in uploading this content. In reusing this media under the specified license, please attribute the creator.

This tag does not indicate the copyright status of the attached work. A normal copyright tag is still required. See Commons:Licensing.

w:en:Creative Commons
attribution share alike
This file is licensed under the Creative Commons Attribution-Share Alike 4.0 International license.
You are free:
  • to share – to copy, distribute and transmit the work
  • to remix – to adapt the work
Under the following conditions:
  • attribution – You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
  • share alike – If you remix, transform, or build upon the material, you must distribute your contributions under the same or compatible license as the original.
Category:CC-BY-SA-4.0#Illustration%20of%20Bayesian%20AB%20test%20analysis.svgCategory:Files uploaded by Wikimedia Foundation staff#Illustration%20of%20Bayesian%20AB%20test%20analysis.svg
Category:Statistical analysis
Category:CC-BY-SA-4.0 Category:Files uploaded by Wikimedia Foundation staff Category:Invalid SVG created with R code Category:Self-published work Category:Statistical analysis