The Bayesian approach to statistical inference and decision analysis may be described in many ways, some alas so simplistic that its subtlety and power can easily be misunderstood.
Mathematically it provides a model of how a rational scientist would update his or her prior knowledge in light of further data or evidence.
Structure and methods
The structure provided by Bayes’ Theorem allows a unified approach to inference, forecasting and decision making. Bayesian methods have been controversial because they involve the explicit introduction of judgement and prior beliefs into the model.
There is no drive to obtain a goal of ‘scientific objectivity’ in the sense of analyses which are completely independent of their participants, viz. independent of the scientist or statistician. Instead, a Bayesian analysis seeks to model explicitly relevant judgements of the participants.
It is openly subjective. By so being, it seeks to allow other scientists to remove the participants’ judgements from the analysis and replace them by their own so that they may see what they would have inferred in the same circumstances with the same data.
Thus for a Bayesian scientific knowledge emerges not from objective analyses but from consensus when a majority of scientists agree that each starting from his or her prior beliefs is led to the same inference on the basis of the data.
Reverend Thomas Bayes
The Rev Thomas Bayes died in 1761, but the school of statistics which bears his name was not really established until the mid 20th century; and it was not established without controversy. But because it provides a clear and consistent approach to all confirmatory statistics and because there are now computational methods that allow its application in almost all practical circumstances, it is now adopted widely.
The International Society for Bayesian Analysis provides links to many resources, including a guide to most a wide range of introductory texts.
Download PDF slides of the presentation 'What are Bayesian methods?'