Statistical distributions and social policy

This New Yorker piece by Malcolm Gladwell is fabulous:

Fifteen years ago, after the Rodney King beating, the Los Angeles Police Department was in crisis. It was accused of racial insensitivity and ill discipline and violence, and the assumption was that those problems had spread broadly throughout the rank and file. In the language of statisticians, it was thought that L.A.P.D.’s troubles had a “normal” distribution—that if you graphed them the result would look like a bell curve, with a small number of officers at one end of the curve, a small number at the other end, and the bulk of the problem situated in the middle. The bell-curve assumption has become so much a part of our mental architecture that we tend to use it to organize experience automatically.

But when the L.A.P.D. was investigated by a special commission headed by Warren Christopher, a very different picture emerged. Between 1986 and 1990, allegations of excessive force or improper tactics were made against eighteen hundred of the eighty-five hundred officers in the L.A.P.D. The broad middle had scarcely been accused of anything. Furthermore, more than fourteen hundred officers had only one or two allegations made against them—and bear in mind that these were not proven charges, that they happened in a four-year period, and that allegations of excessive force are an inevitable feature of urban police work. (The N.Y.P.D. receives about three thousand such complaints a year.) A hundred and eighty-three officers, however, had four or more complaints against them, forty-four officers had six or more complaints, sixteen had eight or more, and one had sixteen complaints. If you were to graph the troubles of the L.A.P.D., it wouldn’t look like a bell curve. It would look more like a hockey stick. It would follow what statisticians call a “power law” distribution—where all the activity is not in the middle but at one extreme.

Gladwell then continues on to describe such issues as problems of compliance, not of policy:

That's what made the findings of the Christopher Commission so unsatisfying. We put together blue-ribbon panels when we’re faced with problems that seem too large for the normal mechanisms of bureaucratic repair. We want sweeping reforms. But what was the commission’s most memorable observation? It was the story of an officer with a known history of doing things like beating up handcuffed suspects who nonetheless received a performance review from his superior stating that he “usually conducts himself in a manner that inspires respect for the law and instills public confidence.” This is what you say about an officer when you haven’t actually read his file, and the implication of the Christopher Commission’s report was that the L.A.P.D. might help solve its problem simply by getting its police captains to read the files of their officers. The L.A.P.D.’s problem was a matter not of policy but of compliance. The department needed to adhere to the rules it already had in place, and that’s not what a public hungry for institutional transformation wants to hear.

And finally, on techniques designed to deal with such distributions:

Power-law solutions have little appeal to the right, because they involve special treatment for people who do not deserve special treatment; and they have little appeal to the left, because their emphasis on efficiency over fairness suggests the cold number-crunching of Chicago-school cost-benefit analysis. Even the promise of millions of dollars in savings or cleaner air or better police departments cannot entirely compensate for such discomfort.

Little appeal to the left? Little appeal to the right? I like what I hear!

But go read the whole article, its excellent.

Share this