Run a mock round

A Type I error is a false positive: rejecting the null hypothesis when it is actually true. Its probability is the significance level alpha, usually 0.05. In a fraud system it means flagging a legitimate transaction.

A Type II error is a false negative: failing to reject the null when a real effect exists. Its probability is beta, and statistical power is 1 minus beta, the chance of detecting a true effect. In a medical test it means missing a real disease.

There is a tradeoff: lowering alpha reduces false positives but increases false negatives for a fixed sample size. Increasing sample size or effect size raises power.

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.2, power=0.8, alpha=0.05)

Choose alpha and power based on which error is more costly, and compute the sample size before the experiment.

The Central Limit Theorem says that the sampling distribution of the sample mean approaches a normal distribution as the sample size grows, regardless of the shape of the population distribution, provided observations are independent and the variance is finite.

means = [np.mean(np.random.exponential(2, 50)) for _ in range(10000)]

Implications: it justifies normal-based confidence intervals and t-tests for means even when the underlying data are skewed, as long as n is reasonably large. The common rule of thumb is n greater than about 30, but heavily skewed or heavy-tailed data need more.

It also explains why the standard error shrinks with the square root of n: quadrupling the sample halves the margin of error. It does not apply directly to small samples, dependent data, or statistics other than means without care.

A confidence interval gives a range of plausible values for a population parameter, along with the uncertainty in the estimate. A 95 percent interval means that if we repeated the study many times, about 95 percent of the intervals we constructed would contain the true parameter.

from scipy import stats
mean, se = data.mean(), stats.sem(data)
ci = stats.t.interval(0.95, len(data)-1, loc=mean, scale=se)

It is not the probability that the true value lies in this specific interval under a frequentist view, though that is the common shortcut. Wider intervals mean more uncertainty, coming from smaller samples or noisier data.

Overlapping intervals between two groups suggest no clear difference, but the correct test compares the difference directly rather than eyeballing overlap. Report the point estimate, the interval and the sample size together so stakeholders can judge practical significance.

Linear regression models a continuous target as a linear combination of predictors plus error: y = b0 + b1*x1 + ... + e. Coefficients are estimated by ordinary least squares, which minimizes the sum of squared residuals, or by maximum likelihood.

from sklearn.linear_model import LinearRegression
model = LinearRegression().fit(X, y)

Assumptions:

• Linearity between predictors and the mean of the target.
• Independence of errors, violated by time series or clustered data.
• Homoscedasticity: constant error variance.
• Normally distributed errors, mainly for inference and intervals.
• Little multicollinearity; check variance inflation factors.
• No influential outliers distorting the fit.

Violations affect inference more than prediction. Diagnose with residual plots, QQ plots and influence measures. Remedies include transformations, robust standard errors and adding interaction or polynomial terms.

Simpson's paradox is when a trend appears in several groups but reverses or disappears when the groups are combined. It happens because a confounding variable is unevenly distributed across groups and is related to the outcome.

Classic example: a treatment appears to have a higher success rate overall, but within each severity level it actually performs better. The treatment group contains more mild cases, which inflates its overall rate.

df.groupby("severity").agg(success=("success", "mean"))

Implications: always examine the relevant subgroups and think about causal structure before aggregating. But do not automatically prefer the disaggregated view either, because conditioning on a collider or a mediator can create its own bias.

The lesson is that the unit of analysis and the adjustment set are modelling decisions that should follow from a causal diagram, not from whichever result looks better.

Imagine we assume a change has no real effect, which is the null hypothesis. The p-value answers: if that were true, how surprising would the data we observed be? A p-value of 0.03 means that under the no-effect assumption we would see a result this extreme about 3 percent of the time by chance alone.

When the p-value is below a threshold, usually 0.05, we call the result statistically significant and reject the null hypothesis. It does not prove the effect is real, large or important; it only says the data are unlikely under the null. A tiny p-value with a trivial effect size can still be meaningless for the business.

Significance also depends on sample size and power. Always report the effect size and a confidence interval alongside the p-value, and remember that many tests increase the chance of a false positive.

Start with a clear hypothesis and a primary metric tied to the business, plus guardrail metrics for regressions.

Key steps:

1. Define the unit of randomization, usually the user, and keep it stable across the test.
2. Compute sample size from the minimum detectable effect, power of 0.8, alpha of 0.05 and baseline variance. Underpowered tests waste traffic.
3. Randomize with a hash so assignment is consistent and balanced.
4. Run for at least one full business cycle, avoiding peeking that inflates false positives.
5. Analyze with the pre-declared metric and test, using intention-to-treat.

treatment = hash(user_id) % 100 < 50

Watch for novelty effects, sample ratio mismatch, and interactions between concurrent tests. Segment analyses are exploratory unless pre-registered. Combine statistics with judgment about practical significance before shipping.

Bayes theorem updates a belief given evidence: P(A|B) = P(B|A) * P(A) / P(B). The prior P(A) is your initial belief, the likelihood P(B|A) is how probable the evidence is under A, and the posterior P(A|B) is the updated belief.

Example: a disease affects 1 percent of people. A test is 99 percent sensitive and has a 5 percent false positive rate. If you test positive, what is the chance you have it?

p_disease = 0.01
sensitivity = 0.99
false_pos = 0.05
p_pos = sensitivity*p_disease + false_pos*(1-p_disease)
p_posterior = sensitivity*p_disease / p_pos
# 0.99*0.01 / (0.99*0.01 + 0.05*0.99) = 0.1667

Even with a positive test the probability is only about 17 percent, because the disease is rare. This base rate effect is why screening uses confirmatory tests and why base rates matter in fraud and spam detection.

Both test hypotheses about means, and the choice hinges on whether the population standard deviation is known and the sample size.

Use a z-test when the population standard deviation is known and the data are normal, or when the sample is large enough that the sample standard deviation is a good estimate thanks to the Central Limit Theorem, typically n greater than 30 or 50.

Use a t-test when the population standard deviation is unknown and the sample is small. The t-distribution has heavier tails that account for the extra uncertainty of estimating the variance from data; as n grows it converges to the normal.

from scipy import stats
stats.ttest_ind(group_a, group_b, equal_var=False)

Other choices matter too: paired tests for matched observations, Welch's t-test when variances differ, and non-parametric tests such as Mann-Whitney for heavily skewed or ordinal data.