Bayesian Inference (TODO)

---

Bayesian Inference

This note is from a statistics point of view and the textbook source is "All of Statistics", chapter 11

Bayesian Philosophy

• Probability, Sample Spaces and Events, Independent Event, Conditional Probability and Bayes Theorem (Statistics)

Postulates:

• Probability describes degree of belief, not limiting frequency
• We can make probability statements about parameters, even though they are fixed constants
• we make inferences about a parameter by producing a probability distribution for . Inferences, such as point estimates and interval estimates, may then be extracted from this distribution.

First postulates means that for example it is “correct” to say that “the probability that Alber Einstein drank a cup of tea on August 1, 1948” is 0.35. This does not refer to frequency but the my strenght of belief that the proposition is true.

The Bayesian Method

Bayesian Inference steps:

1. We choose a probability density ) called the prior distribution that expresses our belief about a parameter before we see any data.
2. We choose a statistical mdoel that reflects our belief about given .
3. After observing data we update our belief and calculate the posterio distribution .

To see how the third step is carried out, first suppose that is discrete and that there is a single, discrete observation .

Since we are treating like a random variable, i will denote it as .

• This is the Bayes Theorem (Statistics)

Considering IID samples

The version for continuos variables is obtained by using probability density functions:

replace with:

• is the likelihood

Then:

• notation for ) (and also
• is a normalization constant gived by:

Posterior is proportional to Likelihood times Prior or in symbols:

The constant isn’t considered and can be recovered later.

With the posterio distribution, we can get a point estimate by summarizing the center of the posterior (typically mean or mode of the posterior)

The posterior Covariance, Variance and Mean|mean is:

We can also obtain a Bayesian interval estimate. We find and such that .

Let , Then:

So is a posterior interval.

Example with Bernoulli

Let Bernoulli(). Suppose we take the uniform distribution as a prior.

By Bayes’ Theorem, the posterior has the form:

where is the number of successes.

Recall that a random variable has a Beta distribution with parameters and if its density is:

We see that the posterior for is a Beta distribution with parameters and . That is:

That we write as:

Notice that we have figured out the normalizing constant without actually doing the integral . The mean of a Beta() distribution is so the Bayes estimator is:

It is instructive to rewrite the estimator as:

• is the MLE
• is the prior mean

A 95 percent posterior interval can be obtained by numerically finding and such that .

Now, suppose that instead of a uniform prior, we use the prior . Repeating the calculations above, you will see that:

The flat prior is just the special case with .

The posterior mean is:

• is the prior mean.

In the previous example, the prior was a Beta distribution and the posterior was a Beta distribution. When the prior and the posterior are in the same family, we say that the prior is conjugate with respect to the model.

Example with Gaussian

Let . Assume we know .

Suppose we take as prior . By doing calculations as in previous example we will obtain that:

• i.e the standard error of the MLE .

This is another example of a conjugate prior.

Since and as we can conclude that for large the posterior is approximately .

Now, say we want to find the interval such that .

We choose and such that: and

We want to find such that:

We know that , so:

implying that and similarly, .

So a 95 percent Bayesian interval is . Since and , the 95 percent Bayesian interval is approximated by: which is the frequentist confidence interval.

Large Sample Properties of Bayes Procedures

We saw in the previous two examples that posterio mean was close to MLE. This is true in greater generality

Theorem Let be the MLE and let . Under appropriate regularity conditions, the posterior is approximately Normal with mean and standard deviation . Hence . Also if is the asymptotic frequentist confidence interval, then is also an approximate Bayesian posterior interval:

There is also a Bayesian delta method. Let . Then:

• where and

Flat Priors, Improper Priors and Noninformative Priors

An important question is: where does one get the prior ? One School of thought called subjectivism says that the prior should refiect our subjective opinion about (before the data are collected). This is possible but impractical in complicated problems especially if there are many parameters.

Moreover, injecting subjective opinion into the analysis is contrary to the goal of making scientific inference as objective as possible.

Noninformative prior An alternative is try to define some sort of noninformative prior, one obvious candidate is the flat prior .

Improper Priors Let with known . Suppose we adopt a flat prior where is a constant. Note that so this is not a prior density in the usual sense. So we call this an improper prior.

But this is not a problem because applying the Bayes theorem and computing the posterior density by multiplting the prior and the likelihood we obtain: . This gives and the resulting point and interval estimaros agree exactly with their frequentist counterparts.

Flat priors are not invariant. Let Bernoulli() and suppose we use the flat prior . This flat prior presumably represents our lack of information about before the experiment. Now let . This is a transformation of and we can compute the resulting distribution for , namely:

which is not flat.

But if we are ignorant about then we are also ignorant about so we should use a fiat prior for . This is a contradiction. Since the notion of a flat prior is not well defined because a flat prior on a parameter does not imply a flat prior on a transformed version of the parameter. Flat priors are not transformation invariant.

Jeffreys’s Prior: it’s a rule for creating priors: where is Fisher information function. This rule turns out to be transformation invariant.

Example

Consider the Bernoulli model. Recall that:

Jeffrey’s rule says to use the prior: . This is a Beta (1/2,1/2) density, that is very close to a uniform density.

In a multiparameter problem, the Jeffrey’s prior is defined to be where denotes the determinant of a matrix and is the Fisher information matrix.

Multiparameter Problems

Bayesian Testing

Strenghts and Weakness of Bayesian Inference