1. Basic Distributions#

From this lecture, make sure you understand what a random variable is, the difference between discrete and continuous distributions, PDF/PMF vs. CDF, and the different types of parameters (shape, scale, rate, location). I’ll slowly expand this list until I’ve got all the distributions we use in the course.

Be careful about what parameterization you’re using, as it will change depending on software. I will try to use Vidakovic [2017] versions here, but may add alternate parameterizations.

Discrete#

Bernoulli Distribution#

  • PMF: \(P(X=k|p) = p^k(1-p)^{1-k}\) for \(k \in \{0, 1\}\)

  • CDF: \(F(k|p) = \begin{cases} 0 & \text{for } k < 0 \\ 1-p & \text{for } 0 \leq k < 1 \\ 1 & \text{for } k \geq 1 \end{cases}\)

  • Mean: \(p\)

  • Variance: \(p(1-p)\)

  • Support: \(\{0, 1\}\)

  • Parameters: \(p\) (probability of success)

  • Notation: \(X \sim Bernoulli(p)\)

Binomial Distribution#

  • PMF: \(P(X=k|n,p) = \binom{n}{k} p^k(1-p)^{n-k}\) for \(k \in \{0, 1, 2, ..., n\}\)

  • CDF: \(F(k|n,p) = \sum_{i=0}^{k} \binom{n}{i} p^i(1-p)^{n-i}\)

  • Mean: \(np\)

  • Variance: \(np(1-p)\)

  • Support: \(\{0, 1, 2, ..., n\}\)

  • Parameters: \(n\) (number of trials), \(p\) (probability of success)

  • Notation: \(X \sim Bin(n, p)\)

Poisson Distribution#

  • PMF: \(P(X=k|\lambda) = e^{-\lambda}\frac{\lambda^k}{k!}\)

  • CDF: \(F(k|\lambda) = e^{-\lambda}\sum_{i=0}^{k} \frac{\lambda^i}{i!}\)

  • Mean: \(\lambda\)

  • Variance: \(\lambda\)

  • Support: \(\{0, 1, 2, ...\}\)

  • Parameters: \(\lambda\) (rate)

  • Notation: \(X \sim Poi(\lambda)\)

Continuous#

Normal Distribution#

  • PDF (variance): \(f(x|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-(x-\mu)^2 / (2\sigma^2)}\)

  • PDF (precision): \(f(x|\mu,\tau) = \sqrt{\frac{\tau}{2\pi}} e^{-(\tau/2) (x-\mu)^2}\)

  • CDF: \(\Phi(x|\mu,\sigma) = \frac{1}{2}[1 + \text{erf}((x-\mu)/(\sigma\sqrt{2}))]\)

  • Mean: \(\mu\)

  • Variance: \(\sigma^2\)

  • Support: \((-\infty, \infty)\)

  • Parameters: \(\mu\) (mean), \(\sigma^2\) (variance), \(\tau\) (precision, defined as \(\tau = 1/\sigma^2\))

  • Notation: \(X \sim N(\mu, \sigma^2)\)

Beta Distribution#

  • PDF: \(f(x|\alpha,\beta) = \frac{x^{\alpha-1} (1-x)^{\beta-1}}{B(\alpha,\beta)}\)

  • CDF: \(I_x(\alpha,\beta)\)

  • Mean: \(\frac{\alpha}{\alpha+\beta}\)

  • Variance: \(\frac{\alpha \beta}{(\alpha+\beta)^2(\alpha+\beta+1)}\)

  • Support: \((0, 1)\)

  • Parameters: \(\alpha,\beta\) (shape parameters)

  • Notation: \(X \sim Be(\alpha, \beta)\)

Cauchy Distribution#

  • PDF: \(f(x|x_0,\gamma) = \frac{1}{\pi\gamma[1 + ((x-x_0)/\gamma)^2]}\)

  • CDF: \(F(x|x_0,\gamma) = \frac{1}{\pi}\arctan\left(\frac{x-x_0}{\gamma}\right) + \frac{1}{2}\)

  • Mean: undefined

  • Variance: undefined

  • Support: \((-\infty, \infty)\)

  • Parameters: \(x_0\) (location), \(\gamma\) (scale)

  • Notation: \(X \sim Cauchy(x_0, \gamma)\)

Exponential Distribution#

  • PDF: \(f(x|\lambda) = \lambda e^{-\lambda x}\)

  • CDF: \(F(x|\lambda) = 1 - e^{-\lambda x}\)

  • Mean: \(\frac{1}{\lambda}\)

  • Variance: \(\frac{1}{\lambda^2}\)

  • Support: \((0, \infty)\)

  • Parameters: \(\lambda\) (rate)

  • Notation: \(X \sim Exp(\lambda)\)

Gamma Distribution#

  • PDF: \(f(x|\alpha,\beta) = \frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}\)

  • CDF: \(F(x|\alpha,\beta) = \frac{\gamma(\alpha, \beta x)}{\Gamma(\alpha)}\)

  • Mean: \(\frac{\alpha}{\beta}\)

  • Variance: \(\frac{\alpha}{\beta^2}\)

  • Support: \((0, \infty)\)

  • Parameters: \(\alpha\) (shape), \(\beta\) (rate)

  • Notation: \(X \sim Ga(\alpha, \beta)\)

Uniform Distribution#

  • PDF: \(f(x|a,b) = \frac{1}{b-a}\)

  • CDF: \(F(x|a,b) = \frac{x-a}{b-a}\)

  • Mean: \(\frac{a+b}{2}\)

  • Variance: \(\frac{(b-a)^2}{12}\)

  • Support: \([a, b]\)

  • Parameters: \(a\) (lower bound), \(b\) (upper bound)

  • Notation: \(X \sim U(a, b)\)

Weibull Distribution#

BUGS#

  • PDF: \(f(x|r, \lambda) = \lambda r x^{r-1} e^{-\lambda x^r}\), for \(x > 0\)

  • CDF: \(F(x|r, \lambda) = 1 - e^{-\lambda x^r}\)

  • Mean: \(\lambda^{-\frac{1}{r}} \Gamma\left(1 + \frac{1}{r}\right)\)

  • Variance: \(\frac{\Gamma(1+2/r) - [\Gamma(1+1/r)]^2}{\lambda^{2/r}}\)

  • Parameters: \(r\) (shape parameter), \(\lambda\) (rate parameter)

  • Support: \((0, \infty)\)

  • Notation: \(X \sim Weibull(r, \lambda)\)

PyMC#

  • PDF: \(f(x|\alpha, \beta) = \frac{\alpha x^{\alpha - 1} e^{-(x/\beta)^{\alpha}}}{\beta^\alpha}\), for \(x > 0\)

  • CDF: \(F(x|\alpha, \beta) = 1 - e^{-(x/\beta)^\alpha}\) for \(x > 0\)

  • Mean: \(\beta \Gamma(1 + \frac{1}{\alpha})\)

  • Variance: \(\beta^2 \Gamma(1+2/\alpha - \mu^2/\beta^2)\)

  • Parameters: \(\alpha\) (shape parameter, \(\alpha > 0\)), \(\beta\) (scale parameter, \(\beta > 0\))

    • \(\alpha = r\)

    • \(\beta = \lambda^{-1/\alpha}\)

  • Notation: \(X \sim Weibull(\alpha, \beta)\)

Other resources#

I highly recommend this overview of probability density functions and families by Michael Betancourt, especially section 2.