6
" Will those insights be tested,or simply used to justify the status quo and reinforce prejudices? When I consider the sloppy and self-serving ways that companies use data, I'm often reminded of phrenology, a pseudoscience that was briefly the rage in the nineteenth century. Phrenologists would run their fingers over the patient's skull, probing for bumps and indentations. Each one, they thought, was linked to personality traits that existed in twenty-seven regions of the brain. Usually the conclusion of the phrenologist jibed with the observations he made. If the patient was morbidly anxious or suffering from alcoholism, the skull probe would usually find bumps and dips that correlated with that observation - which, in turn, bolstered faith in the science of phrenology. Phrenology was a model that relied on pseudoscientific nonsense to make authoritative pronouncements, and for decades it went untested. Big Data can fall into the same trap. Models like the ones that red-lighted Kyle Behm and black-balled foreign medical students and St. George's can lock people out, even when the "science" inside them is little more than a bundle of untested assumptions. "
― Cathy O'Neil , Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
7
" Will those insights be tested, or simply used to justify the status quo and reinforce prejudices? When I consider the sloppy and self-serving ways that companies use data, I'm often reminded of phrenology, a pseudoscience that was briefly the rage in the nineteenth century. Phrenologists would run their fingers over the patient's skull, probing for bumps and indentations. Each one, they thought, was linked to personality traits that existed in twenty-seven regions of the brain. Usually the conclusion of the phrenologist jibed with the observations he made. If the patient was morbidly anxious or suffering from alcoholism, the skull probe would usually find bumps and dips that correlated with that observation - which, in turn, bolstered faith in the science of phrenology. Phrenology was a model that relied on pseudoscientific nonsense to make authoritative pronouncements, and for decades it went untested. Big Data can fall into the same trap. Models like the ones that red-lighted Kyle Behm and black-balled foreign medical students and St. George's can lock people out, even when the "science" inside them is little more than a bundle of untested assumptions. "
― Cathy O'Neil , Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
15
" Racism, at the individual level, can be seen as a predictive model whirring away in billions of human minds around the world. It is built from faulty, incomplete, or generalized data. Whether it comes from experience or hearsay, the data indicates that certain types of people have behaved badly. That generates a binary prediction that all people of that race will behave that same way. Needless to say, racists don’t spend a lot of time hunting down reliable data to train their twisted models. And once their model morphs into a belief, it becomes hardwired. It generates poisonous assumptions, yet rarely tests them, settling instead for data that seems to confirm and fortify them. Consequently, racism is the most slovenly of predictive models. It is powered by haphazard data gathering and spurious correlations, reinforced by institutional inequities, and polluted by confirmation bias. "
― Cathy O'Neil , Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
