A naive estimator

To understand the power of the Kaplan–Meier estimator, it is worthwhile to first describe a naive estimator of the survival function.

Fix ${\displaystyle k\in [n]:=\{1,\dots ,n\}}$ and let ${\displaystyle t>0}$. A basic argument shows that the following proposition holds:

Proposition 1: If the censoring time ${\displaystyle c_{k}}$ of event ${\displaystyle k}$ exceeds ${\displaystyle t}$ (${\displaystyle c_{k}\geq t}$), then ${\displaystyle {\tilde {\tau }}_{k}\geq t}$ if and only if ${\displaystyle \tau _{k}\geq t}$.

Let ${\displaystyle k}$ be such that ${\displaystyle c_{k}\geq t}$. It follows from the above proposition that

${\displaystyle \operatorname {Prob} (\tau _{k}\geq t)=\operatorname {Prob} ({\tilde {\tau }}_{k}\geq t).}$

Let ${\displaystyle X_{k}=\mathbb {I} ({\tilde {\tau }}_{k}\geq t)}$ and consider only those ${\displaystyle k\in C(t):=\{1\leq k\leq n\,:\,c_{k}\geq t\}}$, i.e. the events for which the outcome was not censored before time ${\displaystyle t}$. Let ${\displaystyle m(t)=|C(t)|}$ be the number of elements in ${\displaystyle C(t)}$. Note that the set ${\displaystyle C(t)}$ is not random and so neither is ${\displaystyle m(t)}$. Furthermore, ${\displaystyle (X_{k})_{k\in C(t)}}$ is a sequence of independent, identically distributed Bernoulli random variables with common parameter ${\displaystyle S(t)=\operatorname {Prob} (\tau \geq t)}$. Assuming that ${\displaystyle m(t)>0}$, this suggests to estimate ${\displaystyle S(t)}$ using

${\displaystyle {\hat {S}}_{\text{naive}}(t)={\frac {1}{m(t)}}\sum _{k:c_{k}\geq t}X_{k}={\frac {|\{1\leq k\leq n\,:\,{\tilde {\tau }}_{k}\geq t\}|}{|\{1\leq k\leq n\,:\,c_{k}\geq t\}|}}={\frac {|\{1\leq k\leq n\,:\,{\tilde {\tau }}_{k}\geq t\}|}{m(t)}},}$

where the second equality follows because ${\displaystyle {\tilde {\tau }}_{k}\geq t}$ implies ${\displaystyle c_{k}\geq t}$, while the last equality is simply a change of notation.

The quality of this estimate is governed by the size of ${\displaystyle m(t)}$. This can be problematic when ${\displaystyle m(t)}$ is small, which happens, by definition, when a lot of the events are censored. A particularly unpleasant property of this estimator, that suggests that perhaps it is not the "best" estimator, is that it ignores all the observations whose censoring time precedes ${\displaystyle t}$. Intuitively, these observations still contain information about ${\displaystyle S(t)}$: For example, when for many events with ${\displaystyle c_{k}<t}$, ${\displaystyle {\tilde {\tau }}_{k}<c_{k}}$ also holds, we can infer that events often happen early, which implies that ${\displaystyle \operatorname {Prob} (\tau \leq t)}$ is large, which, through ${\displaystyle S(t)=1-\operatorname {Prob} (\tau \leq t)}$ means that ${\displaystyle S(t)}$ must be small. However, this information is ignored by this naive estimator. The question is then whether there exists an estimator that makes a better use of all the data. This is what the Kaplan–Meier estimator accomplishes. Note that the naive estimator cannot be improved when censoring does not take place; so whether an improvement is possible critically hinges upon whether censoring is in place.

The plug-in approach

By elementary calculations,

${\displaystyle {\begin{aligned}S(t)&=\operatorname {Prob} (\tau >t\mid \tau >t-1)\operatorname {Prob} (\tau >t-1)\\[4pt]&=(1-\operatorname {Prob} (\tau \leq t\mid \tau >t-1))\operatorname {Prob} (\tau >t-1)\\[4pt]&=(1-\operatorname {Prob} (\tau =t\mid \tau \geq t))\operatorname {Prob} (\tau >t-1)\\[4pt]&=q(t)S(t-1)\,,\end{aligned}}}$

where the second to last equality used that ${\displaystyle \tau }$ is integer valued and for the last line we introduced

${\displaystyle q(t)=1-\operatorname {Prob} (\tau =t\mid \tau \geq t).}$

By a recursive expansion of the equality ${\displaystyle S(t)=q(t)S(t-1)}$, we get

${\displaystyle S(t)=q(t)q(t-1)\cdots q(0).}$

Note that here ${\displaystyle q(0)=1-\operatorname {Prob} (\tau =0\mid \tau >-1)=1-\operatorname {Prob} (\tau =0)}$.

The Kaplan–Meier estimator can be seen as a "plug-in estimator" where each ${\displaystyle q(s)}$ is estimated based on the data and the estimator of ${\displaystyle S(t)}$ is obtained as a product of these estimates.

It remains to specify how ${\displaystyle q(s)=1-\operatorname {Prob} (\tau =s\mid \tau \geq s)}$ is to be estimated. By Proposition 1, for any ${\displaystyle k\in [n]}$ such that ${\displaystyle c_{k}\geq s}$, ${\displaystyle \operatorname {Prob} (\tau =s)=\operatorname {Prob} ({\tilde {\tau }}_{k}=s)}$ and ${\displaystyle \operatorname {Prob} (\tau \geq s)=\operatorname {Prob} ({\tilde {\tau }}_{k}\geq s)}$ both hold. Hence, for any ${\displaystyle k\in [n]}$ such that ${\displaystyle c_{k}\geq s}$,

${\displaystyle \operatorname {Prob} (\tau =s|\tau \geq s)=\operatorname {Prob} ({\tilde {\tau }}_{k}=s)/\operatorname {Prob} ({\tilde {\tau }}_{k}\geq s).}$

By a similar reasoning that lead to the construction of the naive estimator above, we arrive at the estimator

${\displaystyle {\hat {q}}(s)=1-{\frac {|\{1\leq k\leq n\,:\,c_{k}\geq s,{\tilde {\tau }}_{k}=s\}|}{|\{1\leq k\leq n\,:\,c_{k}\geq s,{\tilde {\tau }}_{k}\geq s\}|}}=1-{\frac {|\{1\leq k\leq n\,:\,{\tilde {\tau }}_{k}=s\}|}{|\{1\leq k\leq n\,:\,{\tilde {\tau }}_{k}\geq s\}|}}}$

(think of estimating the numerator and denominator separately in the definition of the "hazard rate" ${\displaystyle \operatorname {Prob} (\tau =s|\tau \geq s)}$). The Kaplan–Meier estimator is then given by

${\displaystyle {\hat {S}}(t)=\prod _{s=0}^{t}{\hat {q}}(s).}$

The form of the estimator stated at the beginning of the article can be obtained by some further algebra. For this, write ${\displaystyle {\hat {q}}(s)=1-d(s)/n(s)}$ where, using the actuarial science terminology, ${\displaystyle d(s)=|\{1\leq k\leq n\,:\,\tau _{k}=s\}|}$ is the number of known deaths at time ${\displaystyle s}$, while ${\displaystyle n(s)=|\{1\leq k\leq n\,:\,{\tilde {\tau }}_{k}\geq s\}|}$ is the number of those persons who are alive (and not being censored) at time ${\displaystyle s-1}$.

Note that if ${\displaystyle d(s)=0}$, ${\displaystyle {\hat {q}}(s)=1}$. This implies that we can leave out from the product defining ${\displaystyle {\hat {S}}(t)}$ all those terms where ${\displaystyle d(s)=0}$. Then, letting ${\displaystyle 0\leq t_{1}<t_{2}<\dots <t_{m}}$ be the times ${\displaystyle s}$ when ${\displaystyle d(s)>0}$, ${\displaystyle d_{i}=d(t_{i})}$ and ${\displaystyle n_{i}=n(t_{i})}$, we arrive at the form of the Kaplan–Meier estimator given at the beginning of the article:

${\displaystyle {\hat {S}}(t)=\prod _{i:t_{i}\leq t}\left(1-{\frac {d_{i}}{n_{i}}}\right).}$

As opposed to the naive estimator, this estimator can be seen to use the available information more effectively: In the special case mentioned beforehand, when there are many early events recorded, the estimator will multiply many terms with a value below one and will thus take into account that the survival probability cannot be large.

Derivation as a maximum likelihood estimator

Kaplan–Meier estimator can be derived from maximum likelihood estimation of the discrete hazard function.^[8] More specifically given ${\displaystyle d_{i}}$ as the number of events and ${\displaystyle n_{i}}$ the total individuals at risk at time ${\displaystyle t_{i}}$, discrete hazard rate ${\displaystyle h_{i}}$ can be defined as the probability of an individual with an event at time ${\displaystyle t_{i}}$. Then survival rate can be defined as:

${\displaystyle S(t)=\prod \limits _{i:\ t_{i}\leq t}(1-h_{i})}$

and the likelihood function for the hazard function up to time ${\displaystyle t_{i}}$ is:

${\displaystyle {\mathcal {L}}(h_{j:j\leq i}\mid d_{j:j\leq i},n_{j:j\leq i})=\prod _{j=1}^{i}h_{j}^{d_{j}}(1-h_{j})^{n_{j}-d_{j}}{n_{j} \choose d_{j}}}$

therefore the log likelihood will be:

${\displaystyle \log({\mathcal {L}})=\sum _{j=1}^{i}\left(d_{j}\log(h_{j})+(n_{j}-d_{j})\log(1-h_{j})+\log {n_{j} \choose d_{j}}\right)}$

finding the maximum of log likelihood with respect to ${\displaystyle h_{i}}$ yields:

${\displaystyle {\frac {\partial \log({\mathcal {L}})}{\partial h_{i}}}={\frac {d_{i}}{{\widehat {h}}_{i}}}-{\frac {n_{i}-d_{i}}{1-{\widehat {h}}_{i}}}=0\Rightarrow {\widehat {h}}_{i}={\frac {d_{i}}{n_{i}}}}$

where hat is used to denote maximum likelihood estimation. Given this result, we can write:

${\displaystyle {\widehat {S}}(t)=\prod \limits _{i:\ t_{i}\leq t}\left(1-{\widehat {h}}_{i}\right)=\prod \limits _{i:\ t_{i}\leq t}\left(1-{\frac {d_{i}}{n_{i}}}\right)}$

More generally (for continuous as well as discrete survival distributions), the Kaplan-Meier estimator may be interpreted as a nonparametric maximum likelihood estimator.^[9]