Model - Bootstrap model

Bootstrap testing is a very efficient statistical approach for reading out AB test results

Bootstrap testing is a very efficient statistical approach for reading out AB test results as we explain in this article. While it is indeed quite efficient, the calculation time is relatively high, which makes this test something you might want to use only in specific cases. Because of this large computation time, Datama disables by default the calculation of the significance for each and every subsegment, but you can still re-enable that calculation by toggling the appropriate enabler at the bottom of the page.

Bootstrap testing basically consists in putting all your observations into one bucket, splitting it into two groups and measuring the difference of the means of the two groups. We then replicate that approach multiple times (100 times in Datama) and plot the distribution difference of the mean. This distribution should roughly follow a normal distribution. Placing your actual observation of the difference of the mean on that distribution gives you the answer of the significance of that difference.

See also: wikipedia

Test data: Gsheet