Variety of proofs provided for the different types of convergence of random variables
This article is supplemental for “Convergence of random variables” and provides proofs for selected results.
Several results will be established using the portmanteau lemma: A sequence {Xn} converges in distribution to X if and only if any of the following conditions are met:
-
for all bounded, continuous functions
;
-
for all bounded, Lipschitz functions
;
-
for all closed sets
;
Convergence almost surely implies convergence in probability
[edit]
![{\displaystyle X_{n}\ {\overset {\mathrm {as} }{\rightarrow }}\ X\quad \Rightarrow \quad X_{n}\ {\overset {p}{\rightarrow }}\ X}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d43f61c43266fd67d63489ff4dc06675dfb58413)
Proof: If
converges to
almost surely, it means that the set of points
has measure zero. Now fix
and consider a sequence of sets
![{\displaystyle A_{n}=\bigcup _{m\geq n}\left\{\left|X_{m}-X\right|>\varepsilon \right\}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8c4a65bb16c0854001d8a8425270294c7ebb9dd3)
This sequence of sets is decreasing (
) towards the set
![{\displaystyle A_{\infty }=\bigcap _{n\geq 1}A_{n}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8372fe8bb207ff08bc718975c02bdea20d2aa4ec)
The probabilities of this sequence are also decreasing, so
; we shall show now that this number is equal to zero. Now for any point
outside of
we have
, which implies that
for all
for some
. In particular, for such
the point
will not lie in
, and hence won't lie in
. Therefore,
and so
.
Finally, by continuity from above,
![{\displaystyle \operatorname {Pr} \left(|X_{n}-X|>\varepsilon \right)\leq \operatorname {Pr} (A_{n})\ {\underset {n\to \infty }{\rightarrow }}0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/757d333fb0f3c6250962adf8683d3f05c7112342)
which by definition means that
converges in probability to
.
Convergence in probability does not imply almost sure convergence in the discrete case
[edit]
If Xn are independent random variables assuming value one with probability 1/n and zero otherwise, then Xn converges to zero in probability but not almost surely. This can be verified using the Borel–Cantelli lemmas.
Convergence in probability implies convergence in distribution
[edit]
![{\displaystyle X_{n}\ {\xrightarrow {p}}\ X\quad \Rightarrow \quad X_{n}\ {\xrightarrow {d}}\ X,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f5ebca9be6ad512a51e5305d99722d451e203890)
Proof for the case of scalar random variables
[edit]
Lemma. Let X, Y be random variables, let a be a real number and ε > 0. Then
![{\displaystyle \operatorname {Pr} (Y\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon ).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/643940c61f37f447ebdd8250df70f6766e07f335)
Proof of lemma:
![{\displaystyle {\begin{aligned}\operatorname {Pr} (Y\leq a)&=\operatorname {Pr} (Y\leq a,\ X\leq a+\varepsilon )+\operatorname {Pr} (Y\leq a,\ X>a+\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X\leq a-X,\ a-X<-\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X<-\varepsilon )\\&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (Y-X<-\varepsilon )+\operatorname {Pr} (Y-X>\varepsilon )\\&=\operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon )\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/34b5341511d572d4099a76626f60a56f6dd50711)
Shorter proof of the lemma:
We have
![{\displaystyle {\begin{aligned}\{Y\leq a\}\subset \{X\leq a+\varepsilon \}\cup \{|Y-X|>\varepsilon \}\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/0a621bd5c51ba2458dd11519a3f6ae80952d55d1)
for if
and
, then
. Hence by the union bound,
![{\displaystyle {\begin{aligned}\operatorname {Pr} (Y\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|Y-X|>\varepsilon ).\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/df55375db710d02508d80969bcd9d700b52e2742)
Proof of the theorem: Recall that in order to prove convergence in distribution, one must show that the sequence of cumulative distribution functions converges to the FX at every point where FX is continuous. Let a be such a point. For every ε > 0, due to the preceding lemma, we have:
![{\displaystyle {\begin{aligned}\operatorname {Pr} (X_{n}\leq a)&\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} (|X_{n}-X|>\varepsilon )\\\operatorname {Pr} (X\leq a-\varepsilon )&\leq \operatorname {Pr} (X_{n}\leq a)+\operatorname {Pr} (|X_{n}-X|>\varepsilon )\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/141b22c971cd6f7719e66d20fd3cf3da30388dae)
So, we have
![{\displaystyle \operatorname {Pr} (X\leq a-\varepsilon )-\operatorname {Pr} \left(\left|X_{n}-X\right|>\varepsilon \right)\leq \operatorname {Pr} (X_{n}\leq a)\leq \operatorname {Pr} (X\leq a+\varepsilon )+\operatorname {Pr} \left(\left|X_{n}-X\right|>\varepsilon \right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/704d21732406e05213b27e21467da69067edf076)
Taking the limit as n → ∞, we obtain:
![{\displaystyle F_{X}(a-\varepsilon )\leq \lim _{n\to \infty }\operatorname {Pr} (X_{n}\leq a)\leq F_{X}(a+\varepsilon ),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2cfffb055733b96b7379b69736efe83d9680c623)
where FX(a) = Pr(X ≤ a) is the cumulative distribution function of X. This function is continuous at a by assumption, and therefore both FX(a−ε) and FX(a+ε) converge to FX(a) as ε → 0+. Taking this limit, we obtain
![{\displaystyle \lim _{n\to \infty }\operatorname {Pr} (X_{n}\leq a)=\operatorname {Pr} (X\leq a),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/409c472ecb5d2708fe6d282467cc626e100cf373)
which means that {Xn} converges to X in distribution.
Proof for the generic case
[edit]
The implication follows for when Xn is a random vector by using this property proved later on this page and by taking Yn = X.
Convergence in distribution to a constant implies convergence in probability
[edit]
provided c is a constant.
Proof: Fix ε > 0. Let Bε(c) be the open ball of radius ε around point c, and Bε(c)c its complement. Then
![{\displaystyle \operatorname {Pr} \left(|X_{n}-c|\geq \varepsilon \right)=\operatorname {Pr} \left(X_{n}\in B_{\varepsilon }(c)^{c}\right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/312e5b81e0b21188c0df42c709557edbb34c42bb)
By the portmanteau lemma (part C), if Xn converges in distribution to c, then the limsup of the latter probability must be less than or equal to Pr(c ∈ Bε(c)c), which is obviously equal to zero. Therefore,
![{\displaystyle {\begin{aligned}\lim _{n\to \infty }\operatorname {Pr} \left(\left|X_{n}-c\right|\geq \varepsilon \right)&\leq \limsup _{n\to \infty }\operatorname {Pr} \left(\left|X_{n}-c\right|\geq \varepsilon \right)\\&=\limsup _{n\to \infty }\operatorname {Pr} \left(X_{n}\in B_{\varepsilon }(c)^{c}\right)\\&\leq \operatorname {Pr} \left(c\in B_{\varepsilon }(c)^{c}\right)=0\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8a4e15e0203c32524af744dc7674c20e2a947091)
which by definition means that Xn converges to c in probability.
Convergence in probability to a sequence converging in distribution implies convergence to the same distribution
[edit]
![{\displaystyle |Y_{n}-X_{n}|\ {\xrightarrow {p}}\ 0,\ \ X_{n}\ {\xrightarrow {d}}\ X\ \quad \Rightarrow \quad Y_{n}\ {\xrightarrow {d}}\ X}](https://wikimedia.org/api/rest_v1/media/math/render/svg/814aa1ed410ae7598aea6fef49e89ab6b1e8bb45)
Proof: We will prove this theorem using the portmanteau lemma, part B. As required in that lemma, consider any bounded function f (i.e. |f(x)| ≤ M) which is also Lipschitz:
![{\displaystyle \exists K>0,\forall x,y:\quad |f(x)-f(y)|\leq K|x-y|.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/478815774a02ce83757aecd77305a3abe83dc972)
Take some ε > 0 and majorize the expression |E[f(Yn)] − E[f(Xn)]| as
![{\displaystyle {\begin{aligned}\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X_{n})\right]\right|&\leq \operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\right]\\&=\operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|<\varepsilon \right\}}\right]+\operatorname {E} \left[\left|f(Y_{n})-f(X_{n})\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|\geq \varepsilon \right\}}\right]\\&\leq \operatorname {E} \left[K\left|Y_{n}-X_{n}\right|\mathbf {1} _{\left\{|Y_{n}-X_{n}|<\varepsilon \right\}}\right]+\operatorname {E} \left[2M\mathbf {1} _{\left\{|Y_{n}-X_{n}|\geq \varepsilon \right\}}\right]\\&\leq K\varepsilon \operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|<\varepsilon \right)+2M\operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|\geq \varepsilon \right)\\&\leq K\varepsilon +2M\operatorname {Pr} \left(\left|Y_{n}-X_{n}\right|\geq \varepsilon \right)\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a7895d04947ce7606bd327e92e3c345616ce8c05)
(here 1{...} denotes the indicator function; the expectation of the indicator function is equal to the probability of corresponding event). Therefore,
![{\displaystyle {\begin{aligned}\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X)\right]\right|&\leq \left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X_{n})\right]\right|+\left|\operatorname {E} \left[f(X_{n})\right]-\operatorname {E} \left[f(X)\right]\right|\\&\leq K\varepsilon +2M\operatorname {Pr} \left(|Y_{n}-X_{n}|\geq \varepsilon \right)+\left|\operatorname {E} \left[f(X_{n})\right]-\operatorname {E} \left[f(X)\right]\right|.\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/07da3457b5925c2249e15c8bf3133f0e7861eca1)
If we take the limit in this expression as n → ∞, the second term will go to zero since {Yn−Xn} converges to zero in probability; and the third term will also converge to zero, by the portmanteau lemma and the fact that Xn converges to X in distribution. Thus
![{\displaystyle \lim _{n\to \infty }\left|\operatorname {E} \left[f(Y_{n})\right]-\operatorname {E} \left[f(X)\right]\right|\leq K\varepsilon .}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e286c1b4ccbdffaec69f47d78a52626cba9b680c)
Since ε was arbitrary, we conclude that the limit must in fact be equal to zero, and therefore E[f(Yn)] → E[f(X)], which again by the portmanteau lemma implies that {Yn} converges to X in distribution. QED.
Convergence of one sequence in distribution and another to a constant implies joint convergence in distribution
[edit]
provided c is a constant.
Proof: We will prove this statement using the portmanteau lemma, part A.
First we want to show that (Xn, c) converges in distribution to (X, c). By the portmanteau lemma this will be true if we can show that E[f(Xn, c)] → E[f(X, c)] for any bounded continuous function f(x, y). So let f be such arbitrary bounded continuous function. Now consider the function of a single variable g(x) := f(x, c). This will obviously be also bounded and continuous, and therefore by the portmanteau lemma for sequence {Xn} converging in distribution to X, we will have that E[g(Xn)] → E[g(X)]. However the latter expression is equivalent to “E[f(Xn, c)] → E[f(X, c)]”, and therefore we now know that (Xn, c) converges in distribution to (X, c).
Secondly, consider |(Xn, Yn) − (Xn, c)| = |Yn − c|. This expression converges in probability to zero because Yn converges in probability to c. Thus we have demonstrated two facts:
![{\displaystyle {\begin{cases}\left|(X_{n},Y_{n})-(X_{n},c)\right|\ {\xrightarrow {p}}\ 0,\\(X_{n},c)\ {\xrightarrow {d}}\ (X,c).\end{cases}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1502e43a3efb731e14f82a8b2af0f1359f200343)
By the property proved earlier, these two facts imply that (Xn, Yn) converge in distribution to (X, c).
Convergence of two sequences in probability implies joint convergence in probability
[edit]
![{\displaystyle X_{n}\ {\xrightarrow {p}}\ X,\ \ Y_{n}\ {\xrightarrow {p}}\ Y\ \quad \Rightarrow \quad (X_{n},Y_{n})\ {\xrightarrow {p}}\ (X,Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c086cdd337e1c61ab78aac9b0f3f4826821d2db3)
Proof:
![{\displaystyle {\begin{aligned}\operatorname {Pr} \left(\left|(X_{n},Y_{n})-(X,Y)\right|\geq \varepsilon \right)&\leq \operatorname {Pr} \left(|X_{n}-X|+|Y_{n}-Y|\geq \varepsilon \right)\\&\leq \operatorname {Pr} \left(|X_{n}-X|\geq \varepsilon /2\right)+\operatorname {Pr} \left(|Y_{n}-Y|\geq \varepsilon /2\right)\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c8bacb5e500a0329a17bfffb569683d7bb1808d3)
where the last step follows by the pigeonhole principle and the sub-additivity of the probability measure. Each of the probabilities on the right-hand side converge to zero as n → ∞ by definition of the convergence of {Xn} and {Yn} in probability to X and Y respectively. Taking the limit we conclude that the left-hand side also converges to zero, and therefore the sequence {(Xn, Yn)} converges in probability to {(X, Y)}.