Gaussian Process Regression (GPR) is a non-parametric Bayesian method that places a distribution directly over functions, giving predictions with calibrated uncertainty. Training reduces to fitting a few kernel hyperparameters by maximising the marginal likelihood.

• Notation

1. Choose Kernel

$$ k(\mathbf{x}_i, \mathbf{x}_j) = \sigma_f^2 \exp\left(-\frac{\lVert \mathbf{x}_i - \mathbf{x}_j \rVert^2}{2\ell^2}\right) $$

2. Place Gaussian Process Prior

$$ P(\mathbf{f} \mid \mathbf{X}) = \mathcal{N}(\mathbf{f} \mid \mathbf{0}, \mathbf{K}) $$

3. Relate Observations by Additive Noise

$$ \mathbf{y} = \mathbf{f} + \boldsymbol{\varepsilon}, \qquad \boldsymbol{\varepsilon} \sim \mathcal{N}(\mathbf{0}, \sigma_n^2 \mathbf{I}), \qquad \mathbf{K}_y = \mathbf{K} + \sigma_n^2 \mathbf{I} $$

4. Train by Maximising Log Marginal Likelihood

$$ \boldsymbol{\theta}^* = \arg\max_{\boldsymbol{\theta}} \log P(\mathbf{y} \mid \mathbf{X}) $$

$$ \log P(\mathbf{y} \mid \mathbf{X}) = \underbrace{-\tfrac{1}{2} \mathbf{y}^\top \mathbf{K}y^{-1} \mathbf{y}}{\text{data fit}} - \underbrace{\tfrac{1}{2} \log \lvert \mathbf{K}y \rvert}{\text{complexity penalty}} - \underbrace{\tfrac{n}{2} \log 2\pi}_{\text{constant}} $$

5. Form Test Covariances

$$ \mathbf{K}* = \mathbf{K}(\mathbf{X}, \mathbf{X}), \qquad \mathbf{K}{**} = \mathbf{K}(\mathbf{X}, \mathbf{X}_*) $$

6. Predict Mean and Uncertainty

$$ \bar{\mathbf{f}}* = \mathbf{K}^\top \mathbf{K}y^{-1} \mathbf{y}, \qquad \mathrm{cov}(\mathbf{f}) = \mathbf{K}{**} - \mathbf{K}^\top \mathbf{K}y^{-1} \mathbf{K} $$