Datama Assess plugin aims at assessing statistical significance of observed variations


Overview

Datama Assess extension is composed of three sections:


1. Header section

The header is used for global advanced settings to let you configure and refine your analysis. Read the dedicated section about Settings

  • Documentation: Direct link to this documentation
  • Settings: Refine the settings of the analysis. Find more information on this page


2. Analysis section

Datama propose 3 views for each test-type so as to visualize your data as you can see in the dropdown below

2.1 Views available for each tests

2.1.1 Frequentist Bernoulli

  • Relative variation (For A/B tests)
  • Comparison
  • Density of mean

2.1.2 Bayesian Bernoulli

  • Relative variation (For A/B tests)
  • Comparison
  • Outperforming chances

2.1.3 Volatility

  • Time series
  • Relative variation (For time series)
  • Comparison


2.2 Views description

2.2.1 Density of mean

This slide represents the expected distributions of compared segments (typically AB test variants)

Change the ratio your are analysing by using the dropdown in the title. Within the dropdown KPIs are ordered first KPIs with significant variations and then KPIs with no significance

Datama Assess is set to analyse each steps of your market equation. You can define the market equation in settings

2.2.2 Comparison

Breakdown by dimensions of your choice in the dropdown below, you can sometimes (Frequentist test) show the values of your KPI with confidence intervals. When a difference between start and end is significant the result is highlighted

The dropdown sort dimensions with a significance score which sums the following score for each values (Here, France + Germany + Spain + UK): score = significant * denom * avgdiff. Where ‘significant’ is basically 0 or 1, ‘denom’ is the denominator of the selected kpi and ‘avgdiff’ is the difference between the two ratios.

You can moreover choose to show denominator instead of confidence intervals with the button at the top right corner :

2.2.3 Relative variation (For A/B tests)

Similar to Comparison, this view shows the relative variation sometimes (Frequentist test) with its confidence interval breakdown by dimensions of your choice.

2.2.4 Outperforming chances

Available only for Bayesian test, this view shows percentages of chances that end kpi is greater than start kpi and vice-versa

2.2.5 Time series

Available only for Volatility test, this view shows the evolution of comparisons over time and pinpoint the selected one.

2.2.6 Relative Variation (For time series)

Available only for Volatility test, this view shows same informations than Time series view and you can breakdown by a second dimension.

Each bar represent former typical comparisons and the bigger bar is the selected one.


Depending on your plan the footer can have different informations: