Non-finite (divergent) relative entropy

In general, the support of a matrix M is the orthogonal complement of its kernel, i.e. ${\displaystyle {\text{supp}}(M)={\text{ker}}(M)^{\perp }}$. When considering the quantum relative entropy, we assume the convention that −s · log 0 = ∞ for any s > 0. This leads to the definition that

${\displaystyle S(\rho \|\sigma )=\infty }$

when

${\displaystyle {\text{supp}}(\rho )\cap {\text{ker}}(\sigma )\neq \{0\}.}$

This can be interpreted in the following way. Informally, the quantum relative entropy is a measure of our ability to distinguish two quantum states where larger values indicate states that are more different. Being orthogonal represents the most different quantum states can be. This is reflected by non-finite quantum relative entropy for orthogonal quantum states. Following the argument given in the Motivation section, if we erroneously assume the state ${\displaystyle \rho }$ has support in ${\displaystyle {\text{ker}}(\sigma )}$, this is an error impossible to recover from.

However, one should be careful not to conclude that the divergence of the quantum relative entropy ${\displaystyle S(\rho \|\sigma )}$ implies that the states ${\displaystyle \rho }$ and ${\displaystyle \sigma }$ are orthogonal or even very different by other measures. Specifically, ${\displaystyle S(\rho \|\sigma )}$ can diverge when ${\displaystyle \rho }$ and ${\displaystyle \sigma }$ differ by a vanishingly small amount as measured by some norm. For example, let ${\displaystyle \sigma }$ have the diagonal representation

${\displaystyle \sigma =\sum _{n}\lambda _{n}|f_{n}\rangle \langle f_{n}|}$

with ${\displaystyle \lambda _{n}>0}$ for ${\displaystyle n=0,1,2,\ldots }$ and ${\displaystyle \lambda _{n}=0}$ for ${\displaystyle n=-1,-2,\ldots }$ where ${\displaystyle \{|f_{n}\rangle ,n\in \mathbb {Z} \}}$ is an orthonormal set. The kernel of ${\displaystyle \sigma }$ is the space spanned by the set ${\displaystyle \{|f_{n}\rangle ,n=-1,-2,\ldots \}}$. Next let

${\displaystyle \rho =\sigma +\epsilon |f_{-1}\rangle \langle f_{-1}|-\epsilon |f_{1}\rangle \langle f_{1}|}$

for a small positive number ${\displaystyle \epsilon }$. As ${\displaystyle \rho }$ has support (namely the state ${\displaystyle |f_{-1}\rangle }$) in the kernel of ${\displaystyle \sigma }$, ${\displaystyle S(\rho \|\sigma )}$ is divergent even though the trace norm of the difference ${\displaystyle (\rho -\sigma )}$ is ${\displaystyle 2\epsilon }$ . This means that difference between ${\displaystyle \rho }$ and ${\displaystyle \sigma }$ as measured by the trace norm is vanishingly small as ${\displaystyle \epsilon \to 0}$ even though ${\displaystyle S(\rho \|\sigma )}$ is divergent (i.e. infinite). This property of the quantum relative entropy represents a serious shortcoming if not treated with care.

Corresponding classical statement

For the classical Kullback–Leibler divergence, it can be shown that

${\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}p_{j}\log {\frac {p_{j}}{q_{j}}}\geq 0,}$

and the equality holds if and only if P = Q. Colloquially, this means that the uncertainty calculated using erroneous assumptions is always greater than the real amount of uncertainty.

To show the inequality, we rewrite

${\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}p_{j}\log {\frac {p_{j}}{q_{j}}}=\sum _{j}(-\log {\frac {q_{j}}{p_{j}}})(p_{j}).}$

Notice that log is a concave function. Therefore -log is convex. Applying Jensen's inequality, we obtain

${\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}(-\log {\frac {q_{j}}{p_{j}}})(p_{j})\geq -\log(\sum _{j}{\frac {q_{j}}{p_{j}}}p_{j})=0.}$

Jensen's inequality also states that equality holds if and only if, for all i, q_i = (Σq_j) p_i, i.e. p = q.

The result

Klein's inequality states that the quantum relative entropy

${\displaystyle S(\rho \|\sigma )=\operatorname {Tr} \rho (\log \rho -\log \sigma ).}$

is non-negative in general. It is zero if and only if ρ = σ.

Proof

Let ρ and σ have spectral decompositions

${\displaystyle \rho =\sum _{i}p_{i}v_{i}v_{i}^{*}\;,\;\sigma =\sum _{i}q_{i}w_{i}w_{i}^{*}.}$

So

${\displaystyle \log \rho =\sum _{i}(\log p_{i})v_{i}v_{i}^{*}\;,\;\log \sigma =\sum _{i}(\log q_{i})w_{i}w_{i}^{*}.}$

Direct calculation gives

${\displaystyle S(\rho \|\sigma )=\sum _{k}p_{k}\log p_{k}-\sum _{i,j}(p_{i}\log q_{j})|v_{i}^{*}w_{j}|^{2}}$

${\displaystyle \qquad \quad \;=\sum _{i}p_{i}(\log p_{i}-\sum _{j}\log q_{j}|v_{i}^{*}w_{j}|^{2})}$

${\displaystyle \qquad \quad \;=\sum _{i}p_{i}(\log p_{i}-\sum _{j}(\log q_{j})P_{ij}),}$

where P_{i j} = |v_i*w_j|².

Since the matrix (P_{i j})_{i j} is a doubly stochastic matrix and -log is a convex function, the above expression is

${\displaystyle \geq \sum _{i}p_{i}(\log p_{i}-\log(\sum _{j}q_{j}P_{ij})).}$

Define r_i = Σ_jq_j P_{i j}. Then {r_i} is a probability distribution. From the non-negativity of classical relative entropy, we have

${\displaystyle S(\rho \|\sigma )\geq \sum _{i}p_{i}\log {\frac {p_{i}}{r_{i}}}\geq 0.}$

The second part of the claim follows from the fact that, since -log is strictly convex, equality is achieved in

${\displaystyle \sum _{i}p_{i}(\log p_{i}-\sum _{j}(\log q_{j})P_{ij})\geq \sum _{i}p_{i}(\log p_{i}-\log(\sum _{j}q_{j}P_{ij}))}$

if and only if (P_{i j}) is a permutation matrix, which implies ρ = σ, after a suitable labeling of the eigenvectors {v_i} and {w_i}.