Probability distribution

An overdispersed exponential family of distributions is a generalization of an exponential family and the exponential dispersion model of distributions and includes those families of probability distributions, parameterized by ${\displaystyle {\boldsymbol {\theta }}}$ and ${\displaystyle \tau }$, whose density functions f (or probability mass function, for the case of a discrete distribution) can be expressed in the form

${\displaystyle f_{Y}(\mathbf {y} \mid {\boldsymbol {\theta }},\tau )=h(\mathbf {y} ,\tau )\exp \left({\frac {\mathbf {b} ({\boldsymbol {\theta }})^{\rm {T}}\mathbf {T} (\mathbf {y} )-A({\boldsymbol {\theta }})}{d(\tau )}}\right).\,\!}$

The dispersion parameter, ${\displaystyle \tau }$, typically is known and is usually related to the variance of the distribution. The functions ${\displaystyle h(\mathbf {y} ,\tau )}$, ${\displaystyle \mathbf {b} ({\boldsymbol {\theta }})}$, ${\displaystyle \mathbf {T} (\mathbf {y} )}$, ${\displaystyle A({\boldsymbol {\theta }})}$, and ${\displaystyle d(\tau )}$ are known. Many common distributions are in this family, including the normal, exponential, gamma, Poisson, Bernoulli, and (for fixed number of trials) binomial, multinomial, and negative binomial.

For scalar ${\displaystyle \mathbf {y} }$ and ${\displaystyle {\boldsymbol {\theta }}}$ (denoted ${\displaystyle y}$ and ${\displaystyle \theta }$ in this case), this reduces to

${\displaystyle f_{Y}(y\mid \theta ,\tau )=h(y,\tau )\exp \left({\frac {b(\theta )T(y)-A(\theta )}{d(\tau )}}\right).\,\!}$

${\displaystyle {\boldsymbol {\theta }}}$ is related to the mean of the distribution. If ${\displaystyle \mathbf {b} ({\boldsymbol {\theta }})}$ is the identity function, then the distribution is said to be in canonical form (or natural form). Note that any distribution can be converted to canonical form by rewriting ${\displaystyle {\boldsymbol {\theta }}}$ as ${\displaystyle {\boldsymbol {\theta }}'}$ and then applying the transformation ${\displaystyle {\boldsymbol {\theta }}=\mathbf {b} ({\boldsymbol {\theta }}')}$. It is always possible to convert ${\displaystyle A({\boldsymbol {\theta }})}$ in terms of the new parametrization, even if ${\displaystyle \mathbf {b} ({\boldsymbol {\theta }}')}$ is not a one-to-one function; see comments in the page on exponential families. If, in addition, ${\displaystyle \mathbf {T} (\mathbf {y} )}$ is the identity and ${\displaystyle \tau }$ is known, then ${\displaystyle {\boldsymbol {\theta }}}$ is called the canonical parameter (or natural parameter) and is related to the mean through

${\displaystyle {\boldsymbol {\mu }}=\operatorname {E} (\mathbf {y} )=\nabla A({\boldsymbol {\theta }}).\,\!}$

For scalar ${\displaystyle \mathbf {y} }$ and ${\displaystyle {\boldsymbol {\theta }}}$, this reduces to

${\displaystyle \mu =\operatorname {E} (y)=A'(\theta ).}$

Under this scenario, the variance of the distribution can be shown to be^[2]

${\displaystyle \operatorname {Var} (\mathbf {y} )=\nabla ^{2}A({\boldsymbol {\theta }})d(\tau ).\,\!}$

For scalar ${\displaystyle \mathbf {y} }$ and ${\displaystyle {\boldsymbol {\theta }}}$, this reduces to

${\displaystyle \operatorname {Var} (y)=A''(\theta )d(\tau ).\,\!}$

Linear predictor

The linear predictor is the quantity which incorporates the information about the independent variables into the model. The symbol η (Greek "eta") denotes a linear predictor. It is related to the expected value of the data through the link function.

η is expressed as linear combinations (thus, "linear") of unknown parameters β. The coefficients of the linear combination are represented as the matrix of independent variables X. η can thus be expressed as

${\displaystyle \eta =\mathbf {X} {\boldsymbol {\beta }}.\,}$

Link function

The link function provides the relationship between the linear predictor and the mean of the distribution function. There are many commonly used link functions, and their choice is informed by several considerations. There is always a well-defined canonical link function which is derived from the exponential of the response's density function. However, in some cases it makes sense to try to match the domain of the link function to the range of the distribution function's mean, or use a non-canonical link function for algorithmic purposes, for example Bayesian probit regression.

When using a distribution function with a canonical parameter ${\displaystyle \theta ,}$ the canonical link function is the function that expresses ${\displaystyle \theta }$ in terms of ${\displaystyle \mu ,}$ i.e. ${\displaystyle \theta =b(\mu ).}$ For the most common distributions, the mean ${\displaystyle \mu }$ is one of the parameters in the standard form of the distribution's density function, and then ${\displaystyle b(\mu )}$ is the function as defined above that maps the density function into its canonical form. When using the canonical link function, ${\displaystyle b(\mu )=\theta =\mathbf {X} {\boldsymbol {\beta }},}$ which allows ${\displaystyle \mathbf {X} ^{\rm {T}}\mathbf {Y} }$ to be a sufficient statistic for ${\displaystyle {\boldsymbol {\beta }}}$.

Following is a table of several exponential-family distributions in common use and the data they are typically used for, along with the canonical link functions and their inverses (sometimes referred to as the mean function, as done here).

Common distributions with typical uses and canonical link functions

Distribution	Support of distribution	Typical uses	Link name	Link function, ${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=g(\mu )\,\!}$	Mean function
Normal	real: ${\displaystyle (-\infty ,+\infty )}$	Linear-response data	Identity	${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\mu \,\!}$	${\displaystyle \mu =\mathbf {X} {\boldsymbol {\beta }}\,\!}$
Exponential	real: ${\displaystyle (0,+\infty )}$	Exponential-response data, scale parameters	Negative inverse	${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=-\mu ^{-1}\,\!}$	${\displaystyle \mu =-(\mathbf {X} {\boldsymbol {\beta }})^{-1}\,\!}$
Gamma
Inverse Gaussian	real: ${\displaystyle (0,+\infty )}$		Inverse squared	${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\mu ^{-2}\,\!}$	${\displaystyle \mu =(\mathbf {X} {\boldsymbol {\beta }})^{-1/2}\,\!}$
Poisson	integer: ${\displaystyle 0,1,2,\ldots }$	count of occurrences in fixed amount of time/space	Log	${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\ln(\mu )\,\!}$	${\displaystyle \mu =\exp(\mathbf {X} {\boldsymbol {\beta }})\,\!}$
Bernoulli	integer: ${\displaystyle \{0,1\}}$	outcome of single yes/no occurrence	Logit	${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\ln \left({\frac {\mu }{1-\mu }}\right)\,\!}$	${\displaystyle \mu ={\frac {\exp(\mathbf {X} {\boldsymbol {\beta }})}{1+\exp(\mathbf {X} {\boldsymbol {\beta }})}}={\frac {1}{1+\exp(-\mathbf {X} {\boldsymbol {\beta }})}}\,\!}$
Binomial	integer: ${\displaystyle 0,1,\ldots ,N}$	count of # of "yes" occurrences out of N yes/no occurrences		${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\ln \left({\frac {\mu }{n-\mu }}\right)\,\!}$
Categorical	integer: ${\displaystyle [0,K)}$	outcome of single K-way occurrence		${\displaystyle \mathbf {X} {\boldsymbol {\beta }}=\ln \left({\frac {\mu }{1-\mu }}\right)\,\!}$
	K-vector of integer: ${\displaystyle [0,1]}$, where exactly one element in the vector has the value 1
Multinomial	K-vector of integer: ${\displaystyle [0,N]}$	count of occurrences of different types (1, ..., K) out of N total K-way occurrences

In the cases of the exponential and gamma distributions, the domain of the canonical link function is not the same as the permitted range of the mean. In particular, the linear predictor may be positive, which would give an impossible negative mean. When maximizing the likelihood, precautions must be taken to avoid this. An alternative is to use a noncanonical link function.

In the case of the Bernoulli, binomial, categorical and multinomial distributions, the support of the distributions is not the same type of data as the parameter being predicted. In all of these cases, the predicted parameter is one or more probabilities, i.e. real numbers in the range ${\displaystyle [0,1]}$. The resulting model is known as logistic regression (or multinomial logistic regression in the case that K-way rather than binary values are being predicted).

For the Bernoulli and binomial distributions, the parameter is a single probability, indicating the likelihood of occurrence of a single event. The Bernoulli still satisfies the basic condition of the generalized linear model in that, even though a single outcome will always be either 0 or 1, the expected value will nonetheless be a real-valued probability, i.e. the probability of occurrence of a "yes" (or 1) outcome. Similarly, in a binomial distribution, the expected value is Np, i.e. the expected proportion of "yes" outcomes will be the probability to be predicted.

For categorical and multinomial distributions, the parameter to be predicted is a K-vector of probabilities, with the further restriction that all probabilities must add up to 1. Each probability indicates the likelihood of occurrence of one of the K possible values. For the multinomial distribution, and for the vector form of the categorical distribution, the expected values of the elements of the vector can be related to the predicted probabilities similarly to the binomial and Bernoulli distributions.

Maximum likelihood

The maximum likelihood estimates can be found using an iteratively reweighted least squares algorithm or a Newton's method with updates of the form:

${\displaystyle {\boldsymbol {\beta }}^{(t+1)}={\boldsymbol {\beta }}^{(t)}+{\mathcal {J}}^{-1}({\boldsymbol {\beta }}^{(t)})u({\boldsymbol {\beta }}^{(t)}),}$

where ${\displaystyle {\mathcal {J}}({\boldsymbol {\beta }}^{(t)})}$ is the observed information matrix (the negative of the Hessian matrix) and ${\displaystyle u({\boldsymbol {\beta }}^{(t)})}$ is the score function; or a Fisher's scoring method:

${\displaystyle {\boldsymbol {\beta }}^{(t+1)}={\boldsymbol {\beta }}^{(t)}+{\mathcal {I}}^{-1}({\boldsymbol {\beta }}^{(t)})u({\boldsymbol {\beta }}^{(t)}),}$

where ${\displaystyle {\mathcal {I}}({\boldsymbol {\beta }}^{(t)})}$ is the Fisher information matrix. Note that if the canonical link function is used, then they are the same.^[3]

Bayesian methods

In general, the posterior distribution cannot be found in closed form and so must be approximated, usually using Laplace approximations or some type of Markov chain Monte Carlo method such as Gibbs sampling.

General linear models

A possible point of confusion has to do with the distinction between generalized linear models and general linear models, two broad statistical models. Co-originator John Nelder has expressed regret over this terminology.^[4]

The general linear model may be viewed as a special case of the generalized linear model with identity link and responses normally distributed. As most exact results of interest are obtained only for the general linear model, the general linear model has undergone a somewhat longer historical development. Results for the generalized linear model with non-identity link are asymptotic (tending to work well with large samples).

Linear regression

A simple, very important example of a generalized linear model (also an example of a general linear model) is linear regression. In linear regression, the use of the least-squares estimator is justified by the Gauss–Markov theorem, which does not assume that the distribution is normal.

From the perspective of generalized linear models, however, it is useful to suppose that the distribution function is the normal distribution with constant variance and the link function is the identity, which is the canonical link if the variance is known. Under these assumptions, the least-squares estimator is obtained as the maximum-likelihood parameter estimate.

For the normal distribution, the generalized linear model has a closed form expression for the maximum-likelihood estimates, which is convenient. Most other GLMs lack closed form estimates.

Binary data

When the response data, Y, are binary (taking on only values 0 and 1), the distribution function is generally chosen to be the Bernoulli distribution and the interpretation of μ_i is then the probability, p, of Y_i taking on the value one.

There are several popular link functions for binomial functions.

Multinomial regression

The binomial case may be easily extended to allow for a multinomial distribution as the response (also, a Generalized Linear Model for counts, with a constrained total). There are two ways in which this is usually done:

Count data

Another example of generalized linear models includes Poisson regression which models count data using the Poisson distribution. The link is typically the logarithm, the canonical link.

The variance function is proportional to the mean

${\displaystyle \operatorname {var} (Y_{i})=\tau \mu _{i},\,}$

where the dispersion parameter τ is typically fixed at exactly one. When it is not, the resulting quasi-likelihood model is often described as Poisson with overdispersion or quasi-Poisson.

Correlated or clustered data

The standard GLM assumes that the observations are uncorrelated. Extensions have been developed to allow for correlation between observations, as occurs for example in longitudinal studies and clustered designs:

• Generalized estimating equations (GEEs) allow for the correlation between observations without the use of an explicit probability model for the origin of the correlations, so there is no explicit likelihood. They are suitable when the random effects and their variances are not of inherent interest, as they allow for the correlation without explaining its origin. The focus is on estimating the average response over the population ("population-averaged" effects) rather than the regression parameters that would enable prediction of the effect of changing one or more components of X on a given individual. GEEs are usually used in conjunction with Huber–White standard errors.^[7][8]
• Generalized linear mixed models (GLMMs) are an extension to GLMs that includes random effects in the linear predictor, giving an explicit probability model that explains the origin of the correlations. The resulting "subject-specific" parameter estimates are suitable when the focus is on estimating the effect of changing one or more components of X on a given individual. GLMMs are also referred to as multilevel models and as mixed model. In general, fitting GLMMs is more computationally complex and intensive than fitting GEEs.

Generalized additive models

Generalized additive models (GAMs) are another extension to GLMs in which the linear predictor η is not restricted to be linear in the covariates X but is the sum of smoothing functions applied to the x_is:

${\displaystyle \eta =\beta _{0}+f_{1}(x_{1})+f_{2}(x_{2})+\cdots \,\!}$

The smoothing functions f_i are estimated from the data. In general this requires a large number of data points and is computationally intensive.^[9][10]

Citations

Bibliography