Gaussian Process Regression

Aside from the practical applications of Gaussian processes (“GPs”) and Gaussian process regression (“GPR”) in statistics and machine learning, the theory underlying GPs offers considerable pedagogical value in understanding Bayesian inference, linear regression, kernel methods and stochastic processes in general. GPs if viewed in the right light serve to amplify and possibly unify many of the underlying statistical properties of ML algorithms.

GPs and GPR are often considered an advanced topic in statistics. This may be due to the mathematical elegance in which the topic is typically presented. However, in reality GPR may be understood as a straightforward extension of linear regression in finite spaces to an infinite dimensional space. With that in mind, the intention of this series of articles on GP and GPR will hopefully illuminate GPs and GPR and simultaneously serve as a review of fundamental statistical concepts important machine learning.

Gaussian processes provide some insight into neural networks as Neal showed that starting from a neural network with one hidden layer and letting the number of hidden layers extend to infinity, the properties of the network converge to a Gaussian process (more on this later).

This first article will be brief and simply state the key idea of GPR. Subsequent articles will expand on this introduction in depth and in doing so explore and review basic statistical topics in the context of GPR including:

Multivariate Gaussian Distributions and Their Properties

Stochastic Processes

Kernel Methods

Maximum Likelihood Estimation

Maximum A Posteriori Estimation

Statistical Theory of Linear Regression

Statistical Theory of Bayesian Linear Regression

GPs

GPR

Linear regression is a statistical method for estimating the functional relationship between random variables. It is important to distinguish between linear regression and Bayesian linear regression whereby in the latter, the parameters are integrated out (more on this later). The term “linear” refers to the fact that the model is linear in the parameters. The model may and typically does involve non-linear basis functions such as polynomials, radial basis functions, Fourier basis functions…

Related posts:

When working in an Agile framework, all meetings should have a clear purpose; we want to eliminate waste. Scrum ceremonies are pre-set meetings in an Agile environment. They’re well defined, have…

There has been much wrangling over the pros and cons of moving forward with impeachment proceedings against President Trump (two words that many a conscientious citizen still finds hard to…

Bunga tunggal adalah bunga yang diberikan berdasarakan modal awal dalam satu periode dari awal periode sampai akhir periode atau setiap final jangka waktu tertentu. Misal saat kita menabung di bank…