The Karma of Kaplan-Meier

Our new book, Modelling Mortality with Actuarial Applications, describes several non-parametric estimators of two quantities:

  1. The survival function, \(S_x(t)\), defined as the probability that a person now aged \(x\) will survive at least \(t\) years (\({}_tp_x\) to actuaries), and
  2. The integrated hazard function, \(\Lambda_x(t) = \displaystyle\int_0^t\mu_{x+s}ds\).

The estimators of the above quantities are based on two items of data collected at the times of the observed deaths (denoted by \(t_1,t_2,\ldots,t_n\)):

  1. The number, \(d_{x+t_i}\), who died at time \(t_i\), and
  2. The number, \(l_{x+t_i^-}\), who were alive and under observation immediately before time \(t_i\) (which time we denote by \(t_i^-\)).

Then \(d_{x+t_i}/l_{x+t_i^-}\) is an empirical estimate of the probability of dying at time \(t_i\), conditional on being alive just before time \(t_i\).  This contributes to an estimator of the integrated hazard function and two estimators of \(S_x(t)\):

\[\hat\Lambda_x(t)=\sum_{t_i\leq t}\frac{d_{x+t_i}}{l_{x+t_i^-}}.\qquad(1)\]

  • The Fleming-Harrington estimator of \(S_x(t)\) is:

\[\hat S_x(t)=\exp\left(-\hat\Lambda_x(t)\right).\qquad(2)\]

\[\hat S_x(t)=\prod_{t_i\leq t}\left(1-\frac{d_{x+t_i}}{l_{x+t_i^-}}\right).\qquad(3)\]

It is clear how the Fleming-Harrington estimator is related to the Nelson-Aalen estimator, but where does Kaplan-Meier fit in?  It would be nice to know, not least because the Kaplan-Meier estimator is by far the most commonly used non-parametric estimator in survival modelling (we demonstrate its use with actuarial data in our book).

In my previous blog we found a new representation of the survival function, \(S_x(t)\), as a product integral, which is the middle expression below:

\[S_x(t) = \prod_0^t\left(1-\mu_{x+s}ds\right) = \exp\left(-\int_0^t\mu_{x+s}ds\right).\qquad(4)\]

Essentially, this expresses the multiplicative property of the survival probabilities in the limit as \(ds\to0\).

We often specify a survival model by its hazard rate, \(\mu_{x+s}\), and then expression (4) defines the distribution \(F_x(t)=1-S_x(t)\) of the random future lifetime, \(T_x\), of a person aged \(x\).  We are accustomed to using 'nice' hazard rates, such as parametric functions or constant hazard rates over single years of age.  This 'niceness' confers a corresponding 'niceness' on \(F_x(t)\), namely that it has no 'lumps' of probability.  That is, there is no age \(x+s\) such that \(\Pr[T_x=s]>0\).

However, random variables that take a discrete set of values, such as the binomial or Poisson, are not 'nice' in the sense above — their distribution functions \(F(t)\) add up discrete lumps of probability concentrated on a set of points in the interval \([0,t]\).  We ask, what happens to expression (4) when 'niceness' fails in this way?

The answer is that the product integral is fine, but the exponential formula ceases to be valid.  If a discrete random variable \(T\) has probability mass \(q_i\) at point \(t_i (t_i=t_1,t_2,\ldots)\) then the product integral becomes the discrete product:

\[\prod_{t_i\leq t}\left(1-\frac{q_i}{1-F(t_i^-)}\right)\qquad(5)\]

and this still is equal to the survival function of \(T\).  The fractional term in expression (5) is the conditional probability \(\Pr[T=t_i|T\geq t_i]\), which is the discrete equivalent of the hazard rate  — exactly what is estimated empirically by the ratios \(d_{x+t_i}/l_{x+t_i^-}\) in our data.

Does expression (5) look familiar?  All can now be explained:

  • Expression (4) relates the survival functions and hazard rate in two ways, and is valid for 'nice' hazard rates.
  • The observed death probabilities \(d_{x+t_i}/l_{x+t_i^-}\) are empirical estimates of the discretized hazard rates at times \(t_i\).
  • The Nelson-Aalen estimator is therefore an empirical estimate of the integrated hazard function.
  • The Kaplan-Meier estimator is the product integral of (minus) the Nelson-Aalen estimator (compare expressions (3) and (5)).
  • The Fleming-Harrington estimator is the exponential of (minus) the Nelson-Aalen estimator.

Because the increments of the Nelson-Aalen estimator are all jumps, the exponential equality in expression (4) does not hold.  Therefore, the Kaplan-Meier estimator is not the same as the Fleming-Harrington estimator (although, as we note in our book, the difference is very small indeed for the large data sets typically used by actuaries).  Expression (2) may give the impression that the Fleming-Harrington estimator is the more closely related to the Nelson-Aalen estimator, but it is not.  In fact, it is the product integral in expression (4) that is the more fundamental construction; the exponential formula is just nice to have (if it is valid).

References

Macdonald, A. S., Richards, S. J. and Currie, I. D. (2018) Modelling Mortality with Actuarial Applications, Cambridge University Press (forthcoming).

 

Comments

captcha

Find by key-word


RECENT POSTS

This collection of blogs is called Information Matrix , and ... Read more
Examining residuals is a key aspect of testing a model's ... Read more
Many western countries, including the UK, have recently experienced a ... Read more
Angus Macdonald
Angus Macdonald is a Professor in the School of Mathematical and Computer Sciences at Heriot-Watt University
Nelson-Aalen in Longevitas
Longevitas users can choose whether or not to generate Kaplan-Meier curves with each model fitted.  The default option is to have Kaplan-Meier curves generated, but it can be controlled in the Advanced Options section of the modelling screen.  If the Excel template option is enabled, the Nelson-Aalen curves can also be downloaded from the Chart tab of the model report.