When 2 plus 2 don't equal 4, good data can yield bad conclusions

Once upon a time, in a land of glitz and money, a gambler named Ralph devised a way to win more often by switching bets based on patterns seen in past series of rounds. It applied to craps, baccarat, and roulette which involve wagers on different outcomes – such as Pass or Don't Pass, Player or Banker, and Red or Black in the respective games. Ralph, a prince of a fellow (stories starting with "Once upon a time" usually have princes in them) ran a test to determine whether the system worked. That is, if solid citizens who followed it fared better than those who didn't.

The test involved the three aforementioned games. Other than using the system to decide what propositions to take in any round, players could do anything they pleased. To measure results, bettors were to keep going until they won by doubling their bankrolls or lost by busting out.

All told, 535 stalwarts tried the approach, of whom 240 (44.86 percent) were winners. A casino, eager to ascertain its vulnerability, provided data on 650 regular patrons who played one or another of the games, ignorant of the system. Of these, 289 (44.46 percent) were winners. So a higher fraction of the folks who utilized the method were winners than of those who didn't. Not by much. But casinos live or die on small percentages of many bets so the bosses were worried.

Rosie, an expert on randomness, worked in the casino statistics department. Rosie knew random events exhibit no meaningful patterns so something had to be amiss. She got the raw data for people playing and winning each game with and without the system. Here are the numbers:

with system
game       sessions    winners    pct winners
craps           200         80         40.00%
baccarat        300        146         48.67%
roulette         35         14         40.00%
all             535        240         44.86%
			
			
without system
game       sessions    winners    pct winners
craps           400        170         42.50%
baccarat        200         98         49.00%
roulette         50         21         42.00%
all             650        289         44.46%

Rosie saw that the percentages were, indeed, better for players overall with as opposed to without the system. But she noticed something else. For each game individually, the system made things worse. For craps, 40.00 percent won with while 42.50 percent triumphed without it. For baccarat, 48.67 percent doubled their money with compared to 49.00 percent without the system. And for roulette, 40.00 percent finished on top with whereas 42.00 percent succeeded without it.

How could patrons using the system possibly do worse in every game, but better overall? Rosie investigated and found the conclusion was due to an effect known as Simpson's Paradox. This may occur when data from samples of different sizes are combined and factors that actually cause alternate results are ignored. Its upshot is to falsely invert the relationship between conditions.

Simpson's Paradox goes beyond casinos. In one well-known case, the University of California at Berkeley was accused of gender discrimination in graduate-level admissions. About 44 percent of male and 35 percent of female applicants got into the school. Closer scrutiny showed that within individual departments, rates of admissions by gender were roughly equal. But over half the men had applied for majors having high acceptance rates while about 90 percent of the women sought admission to departments that took lower proportions of candidates. This disparity lead to the false conclusion suggested by the combined figures that males were favored over females.

A more disturbing real example was reported in the American Sociological Review in 1981. A certain judicial district fought charges of bias by claiming its juries were more apt to impose the death penalty for murder on Whites than Blacks. Summary data did show this – 11.88 to 10.24 percent. But when victim's race was considered, the real situation was seen to be otherwise. In each instance, Blacks were sentenced to death more frequently than Whites. Here are the values:

White victim		
defendant's race    # of cases    death penalty    pct getting death penalty
White                      151               19                       12.58%
Black                       63               11                       17.46%

Black victim			
defendant's race    # of cases    death penalty    pct getting death penalty
White                        9                0                        0.00%
Black                      103                6                        5.83%
			
Combined			
defendant's race    # of cases    death penalty    pct getting death penalty
White                      160               19                       11.88%
Black                      166               17                       10.24%

Rosie proved to Ralph that his system hurt rather than helped gamblers, and showed the bosses they had nothing to fear. Rosie got a raise and married Ralph (who was happy to be recognized as an honest man rather then the purveyor of a bogus system), so they and the casino bigwigs all lived happily ever after. The poet, Sumner A Ingmark, consecrated the nuptials with this couplet: