12. Multiple Linear Regression

Multiple linear regression is really an extension of simple linear regression when there are multiple predictors \( x_1, x_2, \ldots, x_k \).

\[\begin{split} \begin{align*} y_i & = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \ldots + \beta_k x_{ik} + \epsilon_i, \quad i = 1 , \ldots , n \\ \epsilon_i & \overset{iid}{\sim} N (0,\sigma^2) \end{align*} \end{split}\]

The compact representation can be written as follows, using matrix form.

\[\begin{split} \mathbf{y} = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}_{n \times 1} \quad \mathbf{X} = \begin{bmatrix} 1 & X_{11} & \dots & X_{1k} \\ 1 & X_{21} & \dots & X_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & X_{n1} & \dots & X_{nk} \end{bmatrix}_{n \times p} \quad \boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \vdots \\ \beta_k \end{bmatrix}_{p \times 1} \quad \boldsymbol{\epsilon} = \begin{bmatrix} \epsilon_1 \\ \vdots \\ \epsilon_n \end{bmatrix}_{n \times 1} \end{split}\]

\[ \mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon} \]

There’s an elegant closed-form solution for finding the \( \hat{\boldsymbol{\beta}} \) estimates in classical statistics:

\[ \hat{\boldsymbol{\beta}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y} \]

The estimates \( \hat{\mathbf{y}} \) can then be found by \( \hat{\mathbf{y}} = \mathbf{X} \hat{\boldsymbol{\beta}} \).

In the Bayesian form of multiple linear regression, we place priors on all \( \beta \)‘s and \( \sigma^2 \).

\[\begin{split} \begin{align*} y_{ij} & \sim N(\mu,\sigma^2) && \text{likelihood} \\ \mu & = \beta_0 + \beta_1 x && \text{deterministic relationship} \\ \beta_j & \sim N(0,\sigma_j^2) && \text{prior: } \beta_j, \space j = 0 \text{ to } k \\ \tau & \sim Ga(a,b) && \text{prior: } \tau \\ \sigma^2 & = 1/\tau && \text{deterministic relationship} \end{align*} \end{split}\]

We typically assume that the \( \beta \)‘s are independent. For non-informative priors, we might set \( \sigma_j^2 \) to something large, like \( 10^3 \) or \( 10^4 \), and the \( a \) and \( b \) parameters in \( \tau \)‘s Gamma distribution to something small, like \( 10^{-3} \).

An example using independent Normal priors on \( \beta \)‘s can be found here: Taste of Cheese.

Other methods for defining priors, such as Zellner’s prior, may help to account for covariance between predictors.

\[\begin{split} \begin{align*} \boldsymbol{\beta} & \sim MVN(\mu,g \cdot \sigma^2 \mathbf{V}) && \text{prior: } \boldsymbol{\beta} \\ \sigma^2 & \sim IG(a,b) && \text{prior: } \sigma^2 \end{align*} \end{split}\]

\[ \text{typical choices: } g = n, \quad g = p^2, \quad g = \max\{n,p^2\} \]

where \( \sigma^2 \mathbf{V} \) is the covariance matrix. An example using Zellner’s prior can be found here: Brozek Index Prediction.

Authors

Jason Naramore, August 2024.