Still another article which equates an educational institution’s success with its graduation rate. (This time: “Black Colleges Need a New Mission” by Jason L. Riley, Wall Street Journal September 28, 2010, p A21). President Obama didn’t make the same mistake in his interview a couple of days ago, but he certainly used graduation rates as a metric for estimating school quality, and other politicians have been more naïve about making the link. And, sad to say, our own university uses graduation rate as an indicator of the success of its component colleges, driven along in this, no doubt, by the rules and opinions of accreditation boards.
There’s always a danger in using any single metric, of course, and graduation rate is a particularly simple and tempting one to collect. But there are two important dangers with respect to its use. First, it gives the wrong answer when comparing disparate populations. (Which hospital is better? One with a mortality rate of 20 in 100 or one with a mortality rate of 1 in 1000? Well, suppose the first is treating patients with advanced cancers and the second is treating nosebleeds…) Second, it gives the wrong incentives. Sticking with the hospital analogy, it encourages institutions to reject cancer patients; if you want to show your charter school is good, make sure to screen for students likely to succeed anyway. But in the case of schools the problem is worse: if schools are rewarded for high pass rates, they have the incentive to lower standards and graduate everyone regardless of achievement. In the hospital case, it’s as if we encouraged the doctors to stick a “cured” sign around the patient’s neck and quickly shove him out the door. Unlike the hospital case, however, it’s a lot easier to hide the fact that the awarded degree is meaningless: harder to go back and check whether the graduate is succeeding in his work than whether the cancer has returned.
I suspect that most careful analyses actually take these difficulties into account: use multiple metrics, adjust suitably, watch out for incentives. I just wish columnists would.