Test picker
Upload a pre-test sample of the metric you plan to test. The picker inspects the distribution's shape, outliers, and skewness, then recommends the test that fits best — before the experiment runs.
Frequently asked questions
Why pre-test data and not the variant-split test data?
By the time you have a variant column, you've already run the experiment. The picker's job is to choose the right test before you start, so it operates on a sample of the metric collected pre-test — typically a historical baseline of the population you'll randomize.
How does the test picker choose a test?
It detects 0/1 columns as binary outcomes and otherwise treats the column as continuous. For binary outcomes it recommends a two-proportion z-test when the sample is large and Fisher's exact test for small samples. For continuous data it recommends Welch's t-test when the distribution is approximately normal, and a rank-based test like Mann-Whitney U when there are heavy tails or noticeable skew.
What if the recommended test is not implemented in abshark?
The picker always recommends the statistically best-fit test, even when abshark doesn't ship that test yet. When that happens the recommendation explains why and offers the closest implemented alternative so you're never stuck.
What diagnostics does the test picker produce?
A single pooled view: sample size, mean, median, standard deviation, skewness, quartiles, a histogram, a box plot with IQR-fence outlier counts, a normal QQ plot, and a Shapiro-Wilk normality result (capped at n=5000, where the CLT covers inference).