From Wikipedia, the free encyclopedia
In probability theory, Hoeffding's lemma is an inequality that bounds the moment-generating function of any bounded random variable.[1] It is named after the Finnish–American mathematical statistician Wassily Hoeffding.
The proof of Hoeffding's lemma uses Taylor's theorem and Jensen's inequality. Hoeffding's lemma is itself used in the proof of McDiarmid's inequality.
Statement of the lemma
[edit]
Let X be any real-valued random variable such that
almost surely, i.e. with probability one. Then, for all
,
![{\displaystyle \mathbb {E} \left[e^{\lambda X}\right]\leq \exp {\Big (}\lambda \mathbb {E} [X]+{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )},}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e6e3a35791718dde26b4d85bb8b4c7403e0c25d6)
or equivalently,
![{\displaystyle \mathbb {E} \left[e^{\lambda (X-\mathbb {E} [X])}\right]\leq \exp {\Big (}{\frac {\lambda ^{2}(b-a)^{2}}{8}}{\Big )}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/685cebd91b82a551af8fb1dd61fe2bc0c74afe21)
Without loss of generality, by replacing
by
, we can assume
, so that
.
Since
is a convex function of
, we have that for all
,
![{\displaystyle e^{\lambda x}\leq {\frac {b-x}{b-a}}e^{\lambda a}+{\frac {x-a}{b-a}}e^{\lambda b}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/30d1e52144c8d7af8888c9929081d93310ca6c60)
So,
![{\displaystyle {\begin{aligned}\mathbb {E} \left[e^{\lambda X}\right]&\leq {\frac {b-\mathbb {E} [X]}{b-a}}e^{\lambda a}+{\frac {\mathbb {E} [X]-a}{b-a}}e^{\lambda b}\\&={\frac {b}{b-a}}e^{\lambda a}+{\frac {-a}{b-a}}e^{\lambda b}\\&=e^{L(\lambda (b-a))},\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1556120d473ffd6db0a1b3ae52d6c0a37cdccdc3)
where
. By computing derivatives, we find
and
.
From the AMGM inequality we thus see that
for all
, and thus, from Taylor's theorem, there is some
such that
![{\displaystyle L(h)=L(0)+hL'(0)+{\frac {1}{2}}h^{2}L''(h\theta )\leq {\frac {1}{8}}h^{2}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1f9929c9924c5a60c82c26ab4bbf480cd9135410)
Thus,
.