Gaussian Processes

A Gaussian process defines a probability distribution over possible functions rather than learning a single fixed equation. Predictions include both an expected value and an estimate of uncertainty.

They are especially useful for small datasets, experimental design and Bayesian optimisation. Their main limitation is computational cost, which grows quickly as the number of training observations increases.