Stats Equations and definitions

Probability

For event $A$, $0\le P(A)\le 1$ and $P(\Omega )=1$. Complement: $P(A^c)=1-P(A).$ Conditional probability: $P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$ Bayes’ rule: $P(\theta \mid x)=\frac{P(x\mid \theta )P(\theta )}{P(x)}.$

Summaries

Sample mean: $\bar{x}=\frac{1}{n}{\sum }_{i=1}^n{x}_i.$ Sample variance: $s^2=\frac{1}{n-1}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2.$ Covariance and correlation: $\mathrm{Cov}(X,Y)=E[(X-E[X])(Y-E[Y])],$ ${\rho }_{XY}=\frac{\mathrm{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}.$

Distributions

A probability mass function gives $P(X=x)$ for discrete $X$. A density $f(x)$ gives continuous probabilities by integration: $P(a\le X\le b)={\int }_a^{b}f(x)dx.$ The Normal distribution has density $f(x)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-(x-\mu {)}^2/(2{\sigma }^2)}.$

Likelihood and loss

For independent data $x_1,\dots ,x_n$, $L(\theta )={\prod }_{i=1}^{n}p(x_i\mid \theta ),\ell (\theta )=\log L(\theta )={\sum }_i\log p(x_i\mid \theta ).$ Many ML losses are negative log-likelihoods, so minimising loss often means maximising statistical plausibility.

Expectation and variance

For a random variable $X$, expectation is the probability-weighted average value:

$$\mathbb{E}[X].$$

Variance measures spread around the mean:

\operatorname{Var}(X)=\mathbb{E}\left[(X-\mathbb{E}[X])^2 ight].

Standard deviation is

$$\sigma =\sqrt{\mathrm{Var}(X)}.$$

Likelihood

Given data $x_1,\dots ,x_n$ and parameter $heta$, the likelihood is

$$L(heta)=p(x_1,\dots ,x_n\mid heta).$$

Statistics often asks: which $heta$ makes the observed data least surprising?

Standard error

For a sample mean based on $n$ independent observations with standard deviation $\sigma$, the standard error is

\operatorname{SE}(ar{X})=rac{\sigma}{\sqrt{n}}.

This is the uncertainty in the estimate of the mean, not the scatter of individual data points. More data reduces standard error like $1/\sqrt{n}$, which is powerful but slow: needing twice the precision costs roughly four times the data.