---

Recall:

• Probability, Sample Spaces and Events, Independent Event, Conditional Probability and Bayes Theorem (Statistics)

Random Variables, Distribution Functions and Probability Functions §

• Source: “All of Statistics - A concise course in statistics”

Random variable is defined as a mapping:

that assigns a real number to each outcome .

Intuition: link sample spaces and events to data

Example:

• Flip a coin 10 times. Let be the number of heads in the sequence then
• More complex: Let be the unit disk. Consider drawing a point at random from . A typical outcome is of the form . Some examples of random variables are: , and , and

It is possible to define considering a random variable and a subset of the real line.

Then

Distribution functions and probability functions §

The cumulative distribution function (or distribution function) or CDF is the function: defined by:

Property: the CDR contains all the information about the random variable.

Theorem 1: Let have CDF and let have CDF . If for all then for all .

Theorem 2: A function mapping the real line to is a CDF for some probability if and only if satisfies the following three conditions:

1. is non-decreasing: implies that
2. is normalized i.e and
3. is right-continuos: for all where

Discrete Probability function (or probability mass function) §

is discrete if it takes countably many values .

The probability function is then defined for as

Properties: Thus

• for all

The CDF of is related to by:

Probability density function (PDF) §

A random variable is continuos if there exists a function such that for all ,

and for every it holds:

The function is called probability density function (PDF). We have that:

and at all points at which is differentiable.

Example Suppose that has a PDF:

Clearly and . A random variable with this density is said to have a Uniform (0,1) distribution. This is meant to capute the idea of choosing a point at random between 0 and 1. The respective CDF is given by:

Continuos random variables can lead to confusion

Note that if is continuos then for every . , this is true only for discrete random variables. For continuos random variables you need to integrate to get a probability. Unlike discrete, a PDF can be bigger than 1 For example: if for and 0 otherwise, then for and so it is a well-defined PDF even though in some places.

Lemma:

4. If is continuos then:

Inverse CDR of quantile function §

Let bea random variable with CDF . The inverse CDF or quantile function is defined by:

• for
• stand for infimum i.e., the smallest value of that satisfy the condition
• is the set of all values of where CDF exceeds

If is strictly increasing and continuos then is the unique real number such that .

Intuition: gives the value of below which a fraction of the distribution falls. It is the value below which approximately of the data lies.

• is the first quartile
• is the median (or second quartile)
• is the third quartile

Example Suppose is discrete and takes values with probabilities

• CDF:
• To find : look for the smallest where that is 3 because so the quantile = 3.

Definition: equal in distribution: two random variable and are equal in distribution if for all . This does not mean that and are equal, rather it means that all probability statements about and will be the same.

Some Important Discrete Random Variables §

Notation

It is traditional to write to indicate that has distribution

Point Mass Distribution §

The Point Mass Distribution (pmf) has a point mass distribution at , written as , if in which case:

• The pmf is for and otherwise

Discrete Uniform Distribution §

Let be a given integer. Suppose that has a probability mass function given by:

We say that has a uniform distribution on .

The Bernoulli Distribution §

The probability function is defined as:

Intuition: Let represent a binary coin flip, and for some . Then

Binomial Distribution §

The Binomial distribution describes the probability of getting exactly successes in independent trials, each with success probability .

• → number of ways to choose successes from trials
• → probability of success
• → probability of failure

Intuition: suppose we have a coin which falls heads up with probability for some . The coin is flip times.

Paremeters

is a parameter i.e a fixed real number. Usually it is unknownw and must be estimated from data; that’s what statistical inference is all about. Statistical inference machine learning

Geometric distribution §

has a geometric distribution with parameter written as if:

Poisson Distribution §

See Poisson Distribution

Some Important Continuos Random Variables §

The Uniform Distribution §

has a Uniform() distribution, written as Uniform() if:

where .

The distribution function is:

Normal (Gaussian) §

has a Normal (or Gaussian) distribution with parameters and denoted by if:

is the center or mean of the distribution and is the spread or variance of the distribution.

The Normal plays an important role in probability and statistics. Many phenomena in nature have approximately Normal distributions.

Central Limit Theorem: the distribution of a sum of random variables can be approximated by a Normal distribution.

Traditionally, the standard Normal random variable is denoted by . The PDF and CDF of a standard normal are denoted by and .

Properties:

• If then
• If , then
• If , are independent, then:
• It follows from the first propert that if then:

Thus we can compute any probabilities we want as long as we can compute the CDF of a standard normal.

Gamma Distribution §

For the Gamma function is defined by . has a Gamma distribution with parameters and denoted by if:

The exponential distribution is just a Gamma( distribution.

If Gamma( are independent, then Gamma()

Cauchy Distribution §

has a distribution with degrees of fredom written if:

The distribution is similar to a Normal but it has thicket tilas. In fact, the Normal corresponds to a with .

The Cauchy Distribution is a special case of the distribution corresponding to . The density is:

The X2 distribution §

has a distribution with degrees of freedom written as if:

If are independent standard Normal random variables then

---

Continuos to this part: Bivariate Distributions, Marginal Distributions, Independent Random Variables, Conditional Distributions, IID Samples