Skip to content

Moment Generating Functions (MGF)

Definition / Introduction

  • The Moment Generating Function (MGF) of a random variable \(X\) is a specific transformation that provides an alternative way to characterize its probability distribution.
  • As the name suggests, the MGF can be used to easily generate the moments of the distribution (like the mean \(E[X]\), \(E[X^2]\), etc.), which describe its shape and properties.
  • It's a powerful theoretical tool, particularly useful for proving results about sums of independent random variables and for identifying distributions.
  • Notation: \(M_X(t)\) or \(M(t)\).

Key Concepts

1. Definition

  • The MGF of a random variable \(X\) is defined as the expected value of \(e^{tX}\), where \(t\) is a real-valued auxiliary variable: $$ M_X(t) = E[e^{tX}] $$
  • Calculation (Discrete): If \(X\) is discrete with PMF \(p(x)\): $$ M_X(t) = \sum_{\text{all } x} e^{tx} p(x) $$
  • Calculation (Continuous): If \(X\) is continuous with PDF \(f(x)\): $$ M_X(t) = \int_{-\infty}^{\infty} e^{tx} f(x) \, dx $$
  • Existence: The MGF exists only if the expected value \(E[e^{tX}]\) is finite for \(t\) in some open interval containing 0 (e.g., for \(-h < t < h\) for some \(h > 0\)). Not all distributions have MGFs (e.g., Cauchy).

2. Why "Moment Generating"? (Key Property 1)

  • The derivatives of the MGF evaluated at \(t = 0\) give the moments of \(X\) about the origin (\(E[X^k]\)).
    • 1st Derivative: \(M'_X(0) = \left. \frac{d}{dt} M_X(t) \right|_{t=0} = E[X]\) (Mean)
    • 2nd Derivative: \(M''_X(0) = \left. \frac{d^2}{dt^2} M_X(t) \right|_{t=0} = E[X^2]\)
    • k-th Derivative: \(M_X^{(k)}(0) = \left. \frac{d^k}{dt^k} M_X(t) \right|_{t=0} = E[X^k]\) (k-th moment about the origin)
  • How it works (Intuition): The Taylor series expansion of \(e^{tX}\) around \(t=0\) is \(1 + tX + \frac{t^2 X^2}{2!} + \frac{t^3 X^3}{3!} + \dots\). Taking the expectation term by term gives \(E[e^{tX}] = 1 + tE[X] + \frac{t^2}{2!}E[X^2] + \dots\). The coefficients of \(\frac{t^k}{k!}\) in the MGF's Taylor series are the moments \(E[X^k]\). Differentiation isolates these coefficients at \(t=0\).
  • This allows calculating the mean and variance (\(Var(X) = E[X^2] - (E[X])^2\)) directly from the MGF.

3. Uniqueness Property (Key Property 2)

  • If two random variables \(X\) and \(Y\) have MGFs \(M_X(t)\) and \(M_Y(t)\) that exist and are equal for all \(t\) in an open interval around 0, then \(X\) and \(Y\) have the same probability distribution.
  • Significance: The MGF (when it exists) uniquely identifies the distribution. If you can show two different processes lead to the same MGF, you've shown they follow the same distribution.

4. MGF of Sums of Independent Variables (Key Property 3)

  • If \(X\) and \(Y\) are independent random variables with MGFs \(M_X(t)\) and \(M_Y(t)\), then the MGF of their sum \(S = X + Y\) is the product of their individual MGFs: $$ M_S(t) = M_{X+Y}(t) = M_X(t) M_Y(t) $$
  • Significance: This property is extremely useful for finding the distribution of sums of independent variables. For example, it can be used to show:
    • Sum of independent Binomials (with same \(p\)) is Binomial.
    • Sum of independent Poissons is Poisson.
    • Sum of independent Gammas (with same rate \(\beta\)) is Gamma.
    • Sum of independent Normals is Normal.

5. MGF of Linear Transformations

  • If \(Y = aX + b\), where \(a\) and \(b\) are constants, then its MGF is related to the MGF of \(X\): $$ M_Y(t) = E[e^{t(aX+b)}] = E[e^{atX} e^{tb}] = e^{tb} E[e^{(at)X}] = e^{tb} M_X(at) $$

Examples of MGFs (Common Distributions)

  • Bernoulli(p): \(M_X(t) = (1-p)e^{t \cdot 0} + pe^{t \cdot 1} = (1-p) + pe^t\)
  • Binomial(n, p): \(M_X(t) = [(1-p) + pe^t]^n\) (Product of n independent Bernoulli MGFs)
  • Poisson(λ): \(M_X(t) = e^{\lambda(e^t - 1)}\)
  • Exponential(λ): \(M_X(t) = \frac{\lambda}{\lambda - t}\), for \(t < \lambda\)
  • Normal(μ, σ²): \(M_X(t) = e^{\mu t + \frac{1}{2}\sigma^2 t^2}\)

Limitations and Alternatives

  • Existence: Not all distributions possess an MGF (e.g., distributions with "heavy tails" like Cauchy or some Pareto distributions).
  • Alternative (Characteristic Function): The Characteristic Function \(\phi_X(t) = E[e^{itX}]\) (where \(i\) is the imaginary unit) always exists for any random variable and also uniquely determines the distribution. It shares similar properties regarding moments and sums of independent variables but requires working with complex numbers.

Connections to Other Topics & Relevance

  • Theoretical Statistics: MGFs are frequently used in mathematical statistics to prove theorems about distributions, particularly those related to sums (like proving the CLT or properties of estimators).
  • Distribution Identification: If you derive the MGF of a complex process and recognize it as the MGF of a known distribution, you've identified the distribution of the process.
  • Calculating Moments: Provides an alternative (sometimes easier) method to calculate higher-order moments compared to direct integration/summation.

Summary

  • The Moment Generating Function (MGF) \(M_X(t) = E[e^{tX}]\) is a transform of a random variable's distribution.
  • Key Properties:
    1. Derivatives at \(t=0\) generate moments: \(M_X^{(k)}(0) = E[X^k]\).
    2. Uniquely determines the distribution (if MGF exists).
    3. MGF of a sum of independent RVs is the product of their MGFs: \(M_{X+Y}(t) = M_X(t)M_Y(t)\).
  • Exists only if \(E[e^{tX}]\) is finite in an interval around \(t=0\).
  • Powerful theoretical tool for analyzing distributions and sums of random variables.

Sources