Proper linear belief functions

For variables X and Y, assume there exists a piece of evidence justifying a normal distribution for variables Y while bearing no opinions for variables X. Also, assume that X and Y are not perfectly linearly related, i.e., their correlation is less than 1. This case involves a mix of an ordinary normal distribution for Y and a vacuous belief function for X. Thus, we represent it using a partially swept matrix as follows:

${\displaystyle M({\vec {X}},Y)=\left[{\begin{array}{*{20}c}{\begin{array}{*{20}c}0\\0\\0\end{array}}&{\begin{array}{*{20}c}\mu _{2}\\0\\\Sigma _{22}\\\end{array}}\end{array}}\right]}$

This is how we could understand the representation. Since we are ignorant on X, we use its swept form and set ${\displaystyle \mu _{1}(\Sigma _{11})^{-1}=0}$ and ${\displaystyle -(\Sigma _{11})^{-1}=0}$. Since the correlation between X and Y is less than 1, the regression coefficient of X on Y approaches to 0 when the variance of X approaches to ∞. Therefore, ${\displaystyle (\Sigma _{11})^{-1}\Sigma _{12}=0}$. Similarly, one can prove that ${\displaystyle \mu _{1}(\Sigma _{11})^{-1}\Sigma _{12}=0}$ and ${\displaystyle \Sigma _{21}(\Sigma _{11})^{-1}\Sigma _{12}=0}$.

Linear equations

Suppose X and Y are two row vectors, and Y = XA + b, where A and b are the coefficient matrices. We represent the equation using a partially swept matrix as follows:

${\displaystyle M({\vec {X}},Y)=\left[{\begin{array}{*{20}c}{\begin{array}{*{20}c}0\\0\\A^{T}\end{array}}&{\begin{array}{*{20}c}b\\A\\0\end{array}}\end{array}}\right]}$

We can understand the representation based on the fact that a linear equation contains two pieces of knowledge: (1) complete ignorance about all variables; and (2) a degenerate conditional distribution of dependent variables given independent variables. Since X is an independent vector in the equation, we are completely ignorant about it. Thus, ${\displaystyle \mu _{1}(\Sigma _{11})^{-1}=0}$ and ${\displaystyle -(\Sigma _{11})^{-1}=0}$. Given X = 0, Y is completely determined to be b. Thus, the conditional mean of Y is b and the conditional variance is 0. Also, the regression coefficient matrix is A.

Note that the knowledge to be represented in linear equations is very close to that in a proper linear belief functions, except that the former assumes a perfect correlation between X and Y while the latter does not. This observation is interesting; it characterizes the difference between partial ignorance and linear equations in one parameter — correlation.

Linear regression models

A linear regression model is a more general and interesting case than previous ones. Suppose X and Y are two vectors and Y = XA + b + E, where A and b are the appropriate coefficient matrices and E is an independent white noise satisfying E ~ N(0, Σ). We represent the model as the following partially swept matrix:

${\displaystyle M({\vec {X}},Y)=\left[{\begin{array}{*{20}c}{\begin{array}{*{20}c}0\\0\\A^{T}\end{array}}&{\begin{array}{*{20}c}b\\A\\\Sigma \end{array}}\end{array}}\right]}$

This linear regression model may be considered as the combination of two pieces of knowledge (see later), one is specified by the linear equation involving three variables X, Y, and E, and the other is a simple normal distribution of E, i.e., E ~ N(0, Σ). Alternatively, one may consider it similar to a linear equation, except that, given X = 0, Y is not completely determined to be b. Instead, the conditional mean of Y is b while the conditional variance is Σ. Note that, in this alternative interpretation, a linear regression model forms a basic building block for knowledge representation and is encoded as one moment matrix. Besides, the noise term E does not appear in the representation. Therefore, it makes the representation more efficient.

From representing the six special cases, we see a clear advantage of the moment matrix representation, i.e., it allows a unified representation for seemingly diverse types of knowledge, including linear equations, joint and conditional distributions, and ignorance. The unification is significant not only for knowledge representation in artificial intelligence but also for statistical analysis and engineering computation. For example, the representation treats the typical logical and probabilistic components in statistics — observations, distributions, improper priors (for Bayesian statistics), and linear equation models — not as separate concepts, but as manifestations of a single concept. It allows one to see the inner connections between these concepts or manifestations and to interplay them for computational purposes.

Marginalization

Marginalization projects a linear belief function into one with fewer variables. Expressed as a moment matrix, it is simply the restriction of a nonswept moment matrix to a submatrix corresponding to the remaining variables. For example, for the joint distribution M(X, Y), its marginal to Y is:

${\displaystyle M^{\downarrow Y}(X,Y)=\left[{\begin{array}{*{20}c}\mu _{2}\\\Sigma _{22}\end{array}}\right]}$

When removing a variable, it is important that the variable has not been swept on in the corresponding moment matrix, i.e., it does not have an arrow sign above the variable. For example, projecting the matrix ${\displaystyle M({\vec {X}},Y)}$ to Y produces:

${\displaystyle M^{\downarrow Y}({\vec {X}},Y)=\left[{\begin{array}{*{20}c}\mu _{2}-\mu _{1}(\Sigma _{11})^{-1}\Sigma _{12}\\\Sigma _{22}-\Sigma _{21}(\Sigma _{11})^{-1}\Sigma _{12}\end{array}}\right]}$

which is not the same linear belief function of Y. However, it is easy to see that removing any or all variables in Y from the partially swept matrix will still produce the correct result — a matrix representing the same function for the remaining variables.

To remove a variable that has been already swept on, we have to reverse the sweeping using partial or full reverse sweepings. Assume ${\displaystyle M({\vec {X}})}$ is a fully swept moment matrix,

${\displaystyle M({\vec {X}})=\left({\begin{array}{*{20}c}{\bar {\mu }}\\{\bar {\Sigma }}\\\end{array}}\right)}$

Then a full reverse sweeping of ${\displaystyle M({\vec {X}})}$ will recover the moment matrix M(X) as follows:

${\displaystyle M(X)=\left({\begin{array}{*{20}c}{-{\bar {\mu }}{\bar {\Sigma }}^{-1}}\\{-{\bar {\Sigma }}^{-1}}\\\end{array}}\right)}$

If a moment matrix is in a partially swept form, say

${\displaystyle M({\vec {X}},Y)=\left[{\begin{array}{*{20}c}{\begin{array}{*{20}c}{{\bar {\mu }}_{1}}\\{{\bar {\Sigma }}_{11}}\\{{\bar {\Sigma }}_{21}}\\\end{array}}&{\begin{array}{*{20}c}{{\bar {\mu }}_{2}}\\{{\bar {\Sigma }}_{12}}\\{{\bar {\Sigma }}_{22}}\\\end{array}}\\\end{array}}\right]}$

its partially reverse sweeping on X is defined as follows:

${\displaystyle M(X,Y)=\left[{\begin{array}{*{20}c}{\begin{array}{*{20}c}{-{\bar {\mu }}_{1}({\bar {\Sigma }}_{11})^{-1}}\\{-({\bar {\Sigma }}_{11})^{-1}}\\{-{\bar {\Sigma }}_{21}({\bar {\Sigma }}_{11})^{-1}}\\\end{array}}&{\begin{array}{*{20}c}{{\bar {\mu }}_{2}-{\bar {\mu }}_{1}({\bar {\Sigma }}_{11})^{-1}{\bar {\Sigma }}_{12}}\\{-({\bar {\Sigma }}_{11})^{-1}{\bar {\Sigma }}_{12}}\\{{\bar {\Sigma }}_{22}-{\bar {\Sigma }}_{21}({\bar {\Sigma }}_{11})^{-1}{\bar {\Sigma }}_{12}}\\\end{array}}\\\end{array}}\right]}$

Reverse sweepings are similar to those of forward ones, except for a sign difference for some multiplications. However, forward and reverse sweepings are opposite operations. It can be easily shown that applying the fully reverse sweeping to ${\displaystyle M({\vec {X}})}$ will recover the initial moment matrix M(X). It can also be proved that applying a partial reverse sweeping on X to the matrix ${\displaystyle M({\vec {X}},Y)}$ will recover the moment matrix M(X,Y). As a matter of fact, Liu^[6] proves that a moment matrix will be recovered through a reverse sweeping after a forward sweeping on the same set of variables. It can be also recovered through a forward sweeping after a reverse sweeping. Intuitively, a partial forward sweeping factorizes a joint into a marginal and a conditional, whereas a partial reverse sweeping multiplies them into a joint.

Combination

According to Dempster’s rule, the combination of belief functions may be expressed as the intersection of focal elements and the multiplication of probability density functions. Liping Liu applies the rule to linear belief functions in particular and obtains a formula of combination in terms of density functions. Later he proves a claim by Arthur P. Dempster and reexpresses the formula as the sum of two fully swept matrices. Mathematically, assume ${\displaystyle M_{1}({\vec {X}})=\left({\begin{array}{*{20}c}{{\bar {\mu }}_{1}}\\{{\bar {\Sigma }}_{1}}\\\end{array}}\right)}$ and ${\displaystyle M_{2}({\vec {X}})=\left({\begin{array}{*{20}c}{{\bar {\mu }}_{2}}\\{{\bar {\Sigma }}_{2}}\\\end{array}}\right)}$ are two LBFs for the same vector of variables X. Then their combination is a fully swept matrix:

${\displaystyle M({\vec {X}})=\left({\begin{array}{*{20}c}{{\bar {\mu }}_{1}+{\bar {\mu }}_{2}}\\{{\bar {\Sigma }}_{1}+{\bar {\Sigma }}_{2}}\\\end{array}}\right)}$

This above equation is often used for multiplying two normal distributions. Here we use it to define the combination of two linear belief functions, which include normal distributions as a special case. Also, note that a vacuous linear belief function (0 swept matrix) is the neutral element for combination. When applying the equation, we need to consider two special cases. First, if two matrices to be combined have different dimensions, then one or both matrices must be vacuously extended, i.e., assuming ignorance on the variables that are no present in each matrix. For example, if M₁(X,Y) and M₂(X,Z) are to be combined, we will first extend them into ${\displaystyle M_{1}(X,Y,{\vec {Z}})}$ and ${\displaystyle M_{2}(X,{\vec {Y}},Z)}$ respectively such that ${\displaystyle M_{1}(X,Y,{\vec {Z}})}$ is ignorant about Z and ${\displaystyle M_{2}(X,{\vec {Y}},Z)}$ is ignorant about Y. The vacuous extension was initially proposed by Kong ^[7] for discrete belief functions. Second, if a variable has zero variance, it will not permit a sweeping operation. In this case, we can pretend the variance to be an extremely small number, say ε, and perform the desired sweeping and combination. We can then apply a reverse sweeping to the combined matrix on the same variable and let ε approaches 0. Since zero variance means complete certainty about a variable, this ε-procedure will vanish ε terms in the final result.

In general, to combine two linear belief functions, their moment matrices must be fully swept. However, one may combine a fully swept matrix with a partially swept one directly if the variables of the former matrix have been all swept on in the later. We can use the linear regression model — Y = XA + b + E — to illustrate the property. As we mentioned, the regression model may be considered as the combination of two pieces of knowledge: one is specified by the linear equation involving three variables X, Y, and E, and the other is a simple normal distribution of E, i.e., E ~ N(0, Σ). Let ${\displaystyle M_{1}({\vec {X}},{\vec {\rm {E}}},Y)=\left[{\begin{array}{*{20}c}0&0&b\\0&0&A\\0&0&I\\{A^{T}}&I&0\\\end{array}}\right]}$ and ${\displaystyle M_{2}({\vec {\rm {E}}})=\left[{\begin{array}{*{20}c}0\\{-\Sigma ^{-1}}\\\end{array}}\right]}$ be their moment matrices respectively. Then the two matrices can be combined directly without sweeping ${\displaystyle M_{1}({\vec {X}},{\vec {\rm {E}}},Y)}$ on Y first. The result of the combination is a partially swept matrix as follows:

${\displaystyle M({\vec {X}},{\vec {\rm {E}}},Y)=\left[{\begin{array}{*{20}c}0&0&b\\0&0&A\\0&{-\Sigma ^{-1}}&I\\{A^{T}}&I&0\\\end{array}}\right]}$

If we apply a reverse sweeping on E and then remove E from the matrix, we will obtain the same representation of the regression model.