Etymology

Neither Pascal nor Huygens used the term "expectation" in its modern sense. In particular, Huygens writes:^[7]

That any one Chance or Expectation to win any thing is worth just such a Sum, as wou'd procure in the same Chance and Expectation at a fair Lay. ... If I expect a or b, and have an equal chance of gaining them, my Expectation is worth (a+b)/2.

More than a hundred years later, in 1814, Pierre-Simon Laplace published his tract "Théorie analytique des probabilités", where the concept of expected value was defined explicitly:^[8]

… this advantage in the theory of chance is the product of the sum hoped for by the probability of obtaining it; it is the partial sum which ought to result when we do not wish to run the risks of the event in supposing that the division is made proportional to the probabilities. This division is the only equitable one when all strange circumstances are eliminated; because an equal degree of probability gives an equal right for the sum hoped for. We will call this advantage mathematical hope.

Random variables with finitely many outcomes

Consider a random variable X with a finite list x₁, ..., x_k of possible outcomes, each of which (respectively) has probability p₁, ..., p_k of occurring. The expectation of X is defined as^[12]

${\displaystyle \operatorname {E} [X]=x_{1}p_{1}+x_{2}p_{2}+\cdots +x_{k}p_{k}.}$

Since the probabilities must satisfy p₁ + ⋅⋅⋅ + p_k = 1, it is natural to interpret E[X] as a weighted average of the x_i values, with weights given by their probabilities p_i.

In the special case that all possible outcomes are equiprobable (that is, p₁ = ⋅⋅⋅ = p_k), the weighted average is given by the standard average. In the general case, the expected value takes into account the fact that some outcomes are more likely than others.

Examples

An illustration of the convergence of sequence averages of rolls of a die to the expected value of 3.5 as the number of rolls (trials) grows

• Let ${\displaystyle X}$ represent the outcome of a roll of a fair six-sided die. More specifically, ${\displaystyle X}$ will be the number of pips showing on the top face of the die after the toss. The possible values for ${\displaystyle X}$ are 1, 2, 3, 4, 5, and 6, all of which are equally likely with a probability of 1/6. The expectation of ${\displaystyle X}$ is

${\displaystyle \operatorname {E} [X]=1\cdot {\frac {1}{6}}+2\cdot {\frac {1}{6}}+3\cdot {\frac {1}{6}}+4\cdot {\frac {1}{6}}+5\cdot {\frac {1}{6}}+6\cdot {\frac {1}{6}}=3.5.}$

If one rolls the die ${\displaystyle n}$ times and computes the average (arithmetic mean) of the results, then as ${\displaystyle n}$ grows, the average will almost surely converge to the expected value, a fact known as the strong law of large numbers.

• The roulette game consists of a small ball and a wheel with 38 numbered pockets around the edge. As the wheel is spun, the ball bounces around randomly until it settles down in one of the pockets. Suppose random variable ${\displaystyle X}$ represents the (monetary) outcome of a $1 bet on a single number ("straight up" bet). If the bet wins (which happens with probability 1/38 in American roulette), the payoff is $35; otherwise the player loses the bet. The expected profit from such a bet will be

${\displaystyle \operatorname {E} [\,{\text{gain from }}\$1{\text{ bet}}\,]=-\$1\cdot {\frac {37}{38}}+\$35\cdot {\frac {1}{38}}=-\${\frac {1}{19}}.}$

That is, the expected value to be won from a $1 bet is −$1/19. Thus, in 190 bets, the net loss will probably be about $10.

Random variables with countably many outcomes

Informally, the expectation of a random variable with a countable set of possible outcomes is defined analogously as the weighted average of all possible outcomes, where the weights are given by the probabilities of realizing each given value. This is to say that

${\displaystyle \operatorname {E} [X]=\sum _{i=1}^{n}x_{i}\,p_{i},}$

where x₁, x₂, ..., x_n are the possible outcomes of the random variable X and p₁, p₂, ... p_n are their corresponding probabilities. In many non-mathematical textbooks, this is presented as the full definition of expected values in this context.^[13]

However, there are some subtleties with infinite summation, so the above formula is not suitable as a mathematical definition. In particular, the Riemann series theorem of mathematical analysis illustrates that the value of certain infinite sums involving positive and negative summands depends on the order in which the summands are given. Since the outcomes of a random variable have no naturally given order, this creates a difficulty in defining expected value precisely.

For this reason, many mathematical textbooks only consider the case that the infinite sum given above converges absolutely, which implies that the infinite sum is a finite number independent of the ordering of summands.^[14] In the alternative case that the infinite sum does not converge absolutely, one says the random variable does not have finite expectation.^[14]

Random variables with density

Now consider a random variable X which has a probability density function given by a function f on the real number line. This means that the probability of X taking on a value in any given open interval is given by the integral of f over that interval. The expectation of X is then given by the integral^[15]

${\displaystyle \operatorname {E} [X]=\int _{-\infty }^{\infty }xf(x)\,dx.}$

A general and mathematically precise formulation of this definition uses measure theory and Lebesgue integration, and the corresponding theory of absolutely continuous random variables is described in the next section. The density functions of many common distributions are piecewise continuous, and as such the theory is often developed in this restricted setting.^[16] For such functions, it is sufficient to only consider the standard Riemann integration. Sometimes continuous random variables are defined as those corresponding to this special class of densities, although the term is used differently by various authors.

Analogously to the countably-infinite case above, there are subtleties with this expression due to the infinite region of integration. Such subtleties can be seen concretely if the distribution of X is given by the Cauchy distribution Cauchy(0, π), so that f(x) = (x² + π²)⁻¹. It is straightforward to compute in this case that

${\displaystyle \int _{a}^{b}xf(x)\,dx=\int _{a}^{b}{\frac {x}{x^{2}+\pi ^{2}}}\,dx={\frac {1}{2}}\ln {\frac {b^{2}+\pi ^{2}}{a^{2}+\pi ^{2}}}.}$

The limit of this expression as a → −∞ and b → ∞ does not exist: if the limits are taken so that a = −b, then the limit is zero, while if the constraint 2a = −b is taken, then the limit is ln(2).

To avoid such ambiguities, in mathematical textbooks it is common to require that the given integral converges absolutely, with E[X] left undefined otherwise.^[17] However, measure-theoretic notions as given below can be used to give a systematic definition of E[X] for more general random variables X.

Arbitrary real-valued random variables

All definitions of the expected value may be expressed in the language of measure theory. In general, if X is a real-valued random variable defined on a probability space (Ω, Σ, P), then the expected value of X, denoted by E[X], is defined as the Lebesgue integral^[18]

${\displaystyle \operatorname {E} [X]=\int _{\Omega }X\,d\operatorname {P} .}$

Despite the newly abstract situation, this definition is extremely similar in nature to the very simplest definition of expected values, given above, as certain weighted averages. This is because, in measure theory, the value of the Lebesgue integral of X is defined via weighted averages of approximations of X which take on finitely many values.^[19] Moreover, if given a random variable with finitely or countably many possible values, the Lebesgue theory of expectation is identical with the summation formulas given above. However, the Lebesgue theory clarifies the scope of the theory of probability density functions. A random variable X is said to be absolutely continuous if any of the following conditions are satisfied:

• there is a nonnegative measurable function f on the real line such that

${\displaystyle {\text{P}}(X\in A)=\int _{A}f(x)\,dx,}$

for any Borel set A, in which the integral is Lebesgue.

• the cumulative distribution function of X is absolutely continuous.
• for any Borel set A of real numbers with Lebesgue measure equal to zero, the probability of X being valued in A is also equal to zero
• for any positive number ε there is a positive number δ such that: if A is a Borel set with Lebesgue measure less than δ, then the probability of X being valued in A is less than ε.

These conditions are all equivalent, although this is nontrivial to establish.^[20] In this definition, f is called the probability density function of X (relative to Lebesgue measure). According to the change-of-variables formula for Lebesgue integration,^[21] combined with the law of the unconscious statistician,^[22] it follows that

${\displaystyle \operatorname {E} [X]\equiv \int _{\Omega }X\,d\operatorname {P} =\int _{\mathbb {R} }xf(x)\,dx}$

for any absolutely continuous random variable X. The above discussion of continuous random variables is thus a special case of the general Lebesgue theory, due to the fact that every piecewise-continuous function is measurable.

The expected value of any real-valued random variable ${\displaystyle X}$ can also be defined on the graph of its cumulative distribution function ${\displaystyle F}$ by a nearby equality of areas. In fact, ${\displaystyle \operatorname {E} [X]=\mu }$ with a real number ${\displaystyle \mu }$ if and only if the two surfaces in the ${\displaystyle x}$-${\displaystyle y}$-plane, described by

${\displaystyle x\leq \mu ,\;\,0\leq y\leq F(x)\quad }$ or ${\displaystyle \quad x\geq \mu ,\;\,F(x)\leq y\leq 1}$

respectively, have the same finite area, i.e. if

${\displaystyle \int _{-\infty }^{\mu }F(x)\,dx=\int _{\mu }^{\infty }{\big (}1-F(x){\big )}\,dx}$

and both improper Riemann integrals converge. Finally, this is equivalent to the representation

${\displaystyle \operatorname {E} [X]=\int _{0}^{\infty }{\big (}1-F(x){\big )}\,dx-\int _{-\infty }^{0}F(x)\,dx,}$

also with convergent integrals.^[23][24]

Infinite expected values

Expected values as defined above are automatically finite numbers. However, in many cases it is fundamental to be able to consider expected values of ±∞. This is intuitive, for example, in the case of the St. Petersburg paradox, in which one considers a random variable with possible outcomes x_i = 2ⁱ, with associated probabilities p_i = 2⁻ⁱ, for i ranging over all positive integers. According to the summation formula in the case of random variables with countably many outcomes, one has

$${\displaystyle \operatorname {E} [X]=\sum _{i=1}^{\infty }x_{i}\,p_{i}=2\cdot {\frac {1}{2}}+4\cdot {\frac {1}{4}}+8\cdot {\frac {1}{8}}+16\cdot {\frac {1}{16}}+\cdots =1+1+1+1+\cdots .}$$

It is natural to say that the expected value equals +∞.

There is a rigorous mathematical theory underlying such ideas, which is often taken as part of the definition of the Lebesgue integral.^[19] The first fundamental observation is that, whichever of the above definitions are followed, any nonnegative random variable whatsoever can be given an unambiguous expected value; whenever absolute convergence fails, then the expected value can be defined as +∞. The second fundamental observation is that any random variable can be written as the difference of two nonnegative random variables. Given a random variable X, one defines the positive and negative parts by X ⁺ = max(X, 0) and X ⁻ = −min(X, 0). These are nonnegative random variables, and it can be directly checked that X = X ⁺ − X ⁻. Since E[X ⁺] and E[X ⁻] are both then defined as either nonnegative numbers or +∞, it is then natural to define:

$${\displaystyle \operatorname {E} [X]={\begin{cases}\operatorname {E} [X^{+}]-\operatorname {E} [X^{-}]&{\text{if }}\operatorname {E} [X^{+}]<\infty {\text{ and }}\operatorname {E} [X^{-}]<\infty$$ ;\\+\infty &{\text{if }}\operatorname {E} [X^{+}]=\infty {\text{ and }}\operatorname {E} [X^{-}]<\infty ;\\-\infty &{\text{if }}\operatorname {E} [X^{+}]<\infty {\text{ and }}\operatorname {E} [X^{-}]=\infty ;\\{\text{undefined}}&{\text{if }}\operatorname {E} [X^{+}]=\infty {\text{ and }}\operatorname {E} [X^{-}]=\infty .\end{cases}}}

According to this definition, E[X] exists and is finite if and only if E[X ⁺] and E[X ⁻] are both finite. Due to the formula |X| = X ⁺ + X ⁻, this is the case if and only if E|X| is finite, and this is equivalent to the absolute convergence conditions in the definitions above. As such, the present considerations do not define finite expected values in any cases not previously considered; they are only useful for infinite expectations.

• In the case of the St. Petersburg paradox, one has X ⁻ = 0 and so E[X] = +∞ as desired.
• Suppose the random variable X takes values 1, −2,3, −4, ... with respective probabilities 6π⁻², 6(2π)⁻², 6(3π)⁻², 6(4π)⁻², .... Then it follows that X ⁺ takes value 2k−1 with probability 6((2k−1)π)⁻² for each positive integer k, and takes value 0 with remaining probability. Similarly, X ⁻ takes value 2k with probability 6(2kπ)⁻² for each positive integer k and takes value 0 with remaining probability. Using the definition for non-negative random variables, one can show that both E[X ⁺] = ∞ and E[X ⁻] = ∞ (see Harmonic series). Hence, in this case the expectation of X is undefined.
• Similarly, the Cauchy distribution, as discussed above, has undefined expectation.

Inequalities

Concentration inequalities control the likelihood of a random variable taking on large values. Markov's inequality is among the best-known and simplest to prove: for a nonnegative random variable X and any positive number a, it states that^[37]

$${\displaystyle \operatorname {P} (X\geq a)\leq {\frac {\operatorname {E} [X]}{a}}.}$$

If X is any random variable with finite expectation, then Markov's inequality may be applied to the random variable |X−E[X]|² to obtain Chebyshev's inequality

$${\displaystyle \operatorname {P} (|X-{\text{E}}[X]|\geq a)\leq {\frac {\operatorname {Var} [X]}{a^{2}}},}$$

where Var is the variance.^[37] These inequalities are significant for their nearly complete lack of conditional assumptions. For example, for any random variable with finite expectation, the Chebyshev inequality implies that there is at least a 75% probability of an outcome being within two standard deviations of the expected value. However, in special cases the Markov and Chebyshev inequalities often give much weaker information than is otherwise available. For example, in the case of an unweighted dice, Chebyshev's inequality says that odds of rolling between 1 and 6 is at least 53%; in reality, the odds are of course 100%.^[38] The Kolmogorov inequality extends the Chebyshev inequality to the context of sums of random variables.^[39]

The following three inequalities are of fundamental importance in the field of mathematical analysis and its applications to probability theory.

• Jensen's inequality: Let f: ℝ → ℝ be a convex function and X a random variable with finite expectation. Then^[40]
  $${\displaystyle f(\operatorname {E} (X))\leq \operatorname {E} (f(X)).}$$

  

Part of the assertion is that the negative part of f(X) has finite expectation, so that the right-hand side is well-defined (possibly infinite). Convexity of f can be phrased as saying that the output of the weighted average of two inputs under-estimates the same weighted average of the two outputs; Jensen's inequality extends this to the setting of completely general weighted averages, as represented by the expectation. In the special case that f(x) = |x|^t/s for positive numbers s < t, one obtains the Lyapunov inequality^[41]

$${\displaystyle \left(\operatorname {E} |X|^{s}\right)^{1/s}\leq \left(\operatorname {E} |X|^{t}\right)^{1/t}.}$$

This can also be proved by the Hölder inequality.^[40] In measure theory, this is particularly notable for proving the inclusion L^s ⊂ L^t of L^p spaces, in the special case of probability spaces.

• Hölder's inequality: if p > 1 and q > 1 are numbers satisfying p ⁻¹ + q ⁻¹ = 1, then
  $${\displaystyle \operatorname {E} |XY|\leq (\operatorname {E} |X|^{p})^{1/p}(\operatorname {E} |Y|^{q})^{1/q}.}$$

  

for any random variables X and Y.^[40] The special case of p = q = 2 is called the Cauchy–Schwarz inequality, and is particularly well-known.^[40]

• Minkowski inequality: given any number p ≥ 1, for any random variables X and Y with E|X|^p and E|Y|^p both finite, it follows that E|X + Y|^p is also finite and^[42]
  $${\displaystyle {\Bigl (}\operatorname {E} |X+Y|^{p}{\Bigr )}^{1/p}\leq {\Bigl (}\operatorname {E} |X|^{p}{\Bigr )}^{1/p}+{\Bigl (}\operatorname {E} |Y|^{p}{\Bigr )}^{1/p}.}$$

  

The Hölder and Minkowski inequalities can be extended to general measure spaces, and are often given in that context. By contrast, the Jensen inequality is special to the case of probability spaces.

Expectations under convergence of random variables

In general, it is not the case that ${\displaystyle \operatorname {E} [X_{n}]\to \operatorname {E} [X]}$ even if ${\displaystyle X_{n}\to X}$ pointwise. Thus, one cannot interchange limits and expectation, without additional conditions on the random variables. To see this, let ${\displaystyle U}$ be a random variable distributed uniformly on ${\displaystyle [0,1].}$ For ${\displaystyle n\geq 1,}$ define a sequence of random variables

${\displaystyle X_{n}=n\cdot \mathbf {1} \left\{U\in \left(0,{\tfrac {1}{n}}\right)\right\},}$

with ${\displaystyle {\mathbf {1} }\{A\}}$ being the indicator function of the event ${\displaystyle A.}$ Then, it follows that ${\displaystyle X_{n}\to 0}$ pointwise. But, ${\displaystyle \operatorname {E} [X_{n}]=n\cdot \operatorname {P} \left(U\in \left[0,{\tfrac {1}{n}}\right]\right)=n\cdot {\tfrac {1}{n}}=1}$ for each ${\displaystyle n.}$ Hence, ${\displaystyle \lim _{n\to \infty }\operatorname {E} [X_{n}]=1\neq 0=\operatorname {E} \left[\lim _{n\to \infty }X_{n}\right].}$

Analogously, for general sequence of random variables ${\displaystyle \{Y_{n}:n\geq 0\},}$ the expected value operator is not ${\displaystyle \sigma }$-additive, i.e.

${\displaystyle \operatorname {E} \left[\sum _{n=0}^{\infty }Y_{n}\right]\neq \sum _{n=0}^{\infty }\operatorname {E} [Y_{n}].}$

An example is easily obtained by setting ${\displaystyle Y_{0}=X_{1}}$ and ${\displaystyle Y_{n}=X_{n+1}-X_{n}}$ for ${\displaystyle n\geq 1,}$ where ${\displaystyle X_{n}}$ is as in the previous example.

A number of convergence results specify exact conditions which allow one to interchange limits and expectations, as specified below.

• Monotone convergence theorem: Let ${\displaystyle \{X_{n}:n\geq 0\}}$ be a sequence of random variables, with ${\displaystyle 0\leq X_{n}\leq X_{n+1}}$ (a.s) for each ${\displaystyle n\geq 0.}$ Furthermore, let ${\displaystyle X_{n}\to X}$ pointwise. Then, the monotone convergence theorem states that ${\displaystyle \lim _{n}\operatorname {E} [X_{n}]=\operatorname {E} [X].}$
  Using the monotone convergence theorem, one can show that expectation indeed satisfies countable additivity for non-negative random variables. In particular, let ${\displaystyle \{X_{i}\}_{i=0}^{\infty }}$ be non-negative random variables. It follows from monotone convergence theorem that
  $${\displaystyle \operatorname {E} \left[\sum _{i=0}^{\infty }X_{i}\right]=\sum _{i=0}^{\infty }\operatorname {E} [X_{i}].}$$

  
• Fatou's lemma: Let ${\displaystyle \{X_{n}\geq 0:n\geq 0\}}$ be a sequence of non-negative random variables. Fatou's lemma states that
  $${\displaystyle \operatorname {E} [\liminf _{n}X_{n}]\leq \liminf _{n}\operatorname {E} [X_{n}].}$$

  
  Corollary. Let ${\displaystyle X_{n}\geq 0}$ with ${\displaystyle \operatorname {E} [X_{n}]\leq C}$ for all ${\displaystyle n\geq 0.}$ If ${\displaystyle X_{n}\to X}$ (a.s), then ${\displaystyle \operatorname {E} [X]\leq C.}$
  Proof is by observing that ${\textstyle X=\liminf _{n}X_{n}}$ (a.s.) and applying Fatou's lemma.
• Dominated convergence theorem: Let ${\displaystyle \{X_{n}:n\geq 0\}}$ be a sequence of random variables. If ${\displaystyle X_{n}\to X}$ pointwise (a.s.), ${\displaystyle |X_{n}|\leq Y\leq +\infty }$ (a.s.), and ${\displaystyle \operatorname {E} [Y]<\infty .}$ Then, according to the dominated convergence theorem,
  • ${\displaystyle \operatorname {E} |X|\leq \operatorname {E} [Y]<\infty }$;
  • ${\displaystyle \lim _{n}\operatorname {E} [X_{n}]=\operatorname {E} [X]}$
  • ${\displaystyle \lim _{n}\operatorname {E} |X_{n}-X|=0.}$
• Uniform integrability: In some cases, the equality ${\displaystyle \lim _{n}\operatorname {E} [X_{n}]=\operatorname {E} [\lim _{n}X_{n}]}$ holds when the sequence ${\displaystyle \{X_{n}\}}$ is uniformly integrable.

Relationship with characteristic function

The probability density function ${\displaystyle f_{X}}$ of a scalar random variable ${\displaystyle X}$ is related to its characteristic function ${\displaystyle \varphi _{X}}$ by the inversion formula:

${\displaystyle f_{X}(x)={\frac {1}{2\pi }}\int _{\mathbb {R} }e^{-itx}\varphi _{X}(t)\,\mathrm {d} t.}$

For the expected value of ${\displaystyle g(X)}$ (where ${\displaystyle g:{\mathbb {R} }\to {\mathbb {R} }}$ is a Borel function), we can use this inversion formula to obtain

${\displaystyle \operatorname {E} [g(X)]={\frac {1}{2\pi }}\int _{\mathbb {R} }g(x)\left[\int _{\mathbb {R} }e^{-itx}\varphi _{X}(t)\,\mathrm {d} t\right]\,\mathrm {d} x.}$

If ${\displaystyle \operatorname {E} [g(X)]}$ is finite, changing the order of integration, we get, in accordance with Fubini–Tonelli theorem,

${\displaystyle \operatorname {E} [g(X)]={\frac {1}{2\pi }}\int _{\mathbb {R} }G(t)\varphi _{X}(t)\,\mathrm {d} t,}$

where

${\displaystyle G(t)=\int _{\mathbb {R} }g(x)e^{-itx}\,\mathrm {d} x}$

is the Fourier transform of ${\displaystyle g(x).}$ The expression for ${\displaystyle \operatorname {E} [g(X)]}$ also follows directly from the Plancherel theorem.