2 - Estimation Theory

Summary

Introduction

We consider a set of independent and identically distributed (i.i.d.) random variables X₁, …, X_n following a probability law P_θ, the form of which is known up to some associated parameter θ, appearing as a constant. Both the X_i and θ can be vector-valued. The set X of possible values of the sample X₁, …, X_n is called the sample space. We assume that θ ∈ Θ, the parameter space, so that a parametric family of probability laws may be represented as P_θ = {P_θ; θ ∈ Θ}. One of the objectives of statistical estimation theory is to develop methods of choosing appropriate statistics T_n = T(X₁, …, X_n), that is, functions of the sample observations, to estimate θ (i.e., to guess the true value of θ) in a reproducible way. In this context, T_n is an estimator of θ.

In an alternative (nonparametric) setup, we may allow the probability law P to be a member of a general class P, not necessarily indexed by some parameter, and then our interest lies in estimating the probability law P itself or some functional θ(P) thereof, without specifying the form of P.

There may be many variations of this setup wherein the X_i may not be independent nor identically distributed as in multisample models, linear models, time-series models, stochastic processes, or unequal probability sampling models.