---

Maximum Likelihood Principle and Maximum Likelihood Estimator (MLE)

• ML - Lezione 9 - Parameters Update, Online and Offline Learning, MLP training, Maximum Likelihood Principle and Multinomial Joint Distribution, Softmax function
• ML - Lezione 24 - Expectation and Maximization Algorithm (Clustering), Maximum Loglikelihood Problem

Maximum Likelihood Estimator (MLE)

Suppose we have a sample of independent observations , from some distributions with probability density function , where is an unknown parameter.

The likelihood function is defined as:

A maximum likelihood estimator is a value that maximizes the likelihood:

The MLE is the parameter that makes the observed data most probable. Intuitively, if nature “whispers” a sample to you, you’d pick the parameter under which the universe would most likely have spoken that sample.

From a more philosophical angle, consider that probabilities encode our beliefs about the world. Observing data should update our beliefs; maximizing the likelihood chooses the parameter that is most believable given the evidence, a natural and principled strategy.

Likelihood could be seen as a score of plausability: by the definition of conditional probability, the likelihood function measures how consistent each is with the observed data. Maximizing it is equivalent to choosing the parameter that best explains the observed evidence.

MLE minimizes information Loss

MLE minimizes information Loss. This can be theoretically justified considering the KL divergence.

Consider two distributions:

• the true distributions we want to approximate
• our model

the KL divergence for the two is:

Minimizing this KL divegence is equivalent to losing the least amount of information when approximating with .

Consider the expected log-likelihood under the true distribution :

notice that we can put this into the previous one and obtain:

• The first term is a constant i.e it does not depend from therefore

Thus maximizing the expected log-likelihood is equivalent to minimize the KL divergence. We are minimizing the KL divergence between the true distribution and our approximation therefore reducing the “error” respect to the true distribution, as the error approach 0 it will converge to the true distribution.

The number of data is limited, plus we don’t know explicitly. We have that .

By the “law of large numbers”:

• as

Thus the empirical log-likelihood converges to the expected log-likelihood which in turn minimizes KL divergences.