Credit Risk Modeling Using Excel and VBA with DVD

8

Validation of Rating Systems

Having set up a rating system, it is natural that one wants to assess its quality. There are two dimensions along which ratings are commonly assessed: discrimination and calibration. In checking discrimination, we ask: How well does a rating system rank borrowers according to their true probability of default (PD)? When examining calibration we ask: How well do estimated PDs match true PDs?

The following example illustrates that the two dimensions capture different aspects of rating quality:

Rating system 1 might represent an agency rating system, with A being the best rating. An agency rating itself is not a PD but can be associated with PDs based on average historical default rates per rating class (see Chapter 3). Rating system 2 might be based on a statistical credit scoring model (see Chapter 1) which directly produces PD estimates. The rank ordering of system 1 is perfect, but the PDs differ dramatically from the true ones. By contrast, the average PD of rating system 2 exactly matches the average true PD, and individual deviations from the average PD are small. However, it does not discriminate at all because the system's PDs are inversely related to the true PDs.

The literature has proposed various methods that test for discrimination, calibration or both. There are several reasons why one would want to test for only one aspect of rating quality ...