Q: What is a confidence interval in A/B testing?

A confidence interval is a range of plausible values for the true effect size (such as conversion lift). A 95% confidence interval means that if the experiment were repeated many times, 95% of such intervals would contain the true effect.

Q: Is AB SHARK free to use?

Yes. AB SHARK is a free, open-source A/B test analyzer. There's no signup, no account, and no usage limits. You can run as many analyses as you want directly in your browser.

Question 1

What is statistical significance in an A/B test?

Accepted Answer

Statistical significance means the difference between your control and variant is unlikely to be due to random chance. It's typically measured with a p-value: a p-value below your chosen threshold (commonly 0.05) indicates the result is statistically significant.

Question 2

How is the p-value calculated for a conversion-rate A/B test?

Accepted Answer

For binary conversion data, AB SHARK uses a two-proportion z-test by default. The test statistic compares the observed difference in conversion rates against the standard error under the null hypothesis of equal rates, and the p-value is the tail probability of seeing a result at least as extreme.

Question 3

When should I stop my A/B test?

Accepted Answer

Decide your sample size before you start using a sample-size calculator, then run until you reach it. Peeking at results and stopping early when a test crosses p < 0.05 inflates your false-positive rate. Use AB SHARK's planner to compute the required n given your baseline rate, MDE, alpha, and power.

Question 4

What is a confidence interval in A/B testing?

Accepted Answer

A confidence interval is a range of plausible values for the true effect size (such as conversion lift). A 95% confidence interval means that if the experiment were repeated many times, 95% of such intervals would contain the true effect.

Question 5

Is AB SHARK free to use?

Accepted Answer

Yes. AB SHARK is a free, open-source A/B test analyzer. There's no signup, no account, and no usage limits. You can run as many analyses as you want directly in your browser.

Role	Name	Visitors (n)	Successes
Control
Treatment 1

Analyze experiment

Variants