Nick Bostrom QUOTES

101 " Our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

102 " Conversely, with sufficiently advanced scanning technology and abundant computing power, it might be possible to brute-force and emulation even with a fairly limited understanding of the brain. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

103 " Common sense and natural language understanding have also turned out to be difficult. It is now often thought that achieving a fully human-level performance on these tasks is an “AI-complete” problem, meaning that the difficulty of solving these problems is essentially equivalent to the difficulty of building generally human-level intelligent machines.61 In other words, if somebody were to succeed in creating an AI that could understand natural language as well as a human adult, they would in all likelihood also either already have succeeded in creating an AI that could do everything else that human intelligence can do, or they would be but a very short step from such a general capability. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

104 " If some day we build machine brains that surpass human brains in general intelligence, then this new superintelligence could become very powerful. And, as the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species would depend on the actions of the machine superintelligence. We do have one advantage: we get to build the stuff. In principle, we could build a kind of superintelligence that would protect human values. We would certainly have strong reason to do so. In practice, the control problem—the problem of how to control what the superintelligence would do—looks quite difficult. It also looks like we will only get one chance. Once unfriendly superintelligence exists, it would prevent us from replacing it or changing its preferences. Our fate would be sealed. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

105 " However, setting aside the question of how modernity’s shortcomings stack up against the not-so-inconsiderable failings of earlier epochs, nothing in our definition of collective superintelligence implies that a society with greater collective intelligence is necessarily better off. The definition does not even imply that the more collectively intelligent society is wiser. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

106 " Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

107 " This strategy is exemplified by the sea squirt larva, which swims about until it finds a suitable rock, to which it then permanently affixes itself. Cemented in place, the larva has less need for complex information processing, whence it proceeds to digest part of its own brain (its cerebral ganglion). One can observe the same phenomenon in some academics when they have been granted tenure. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

108 " The traditional illustration of the direct rule-based approach is the “three laws of robotics” concept, formulated by science fiction author Isaac Asimov in a short story published in 1942.22 The three laws were: (1) A robot may not injure a human being or, through inaction, allow a human being to come to harm; (2) A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law; (3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. Embarrassingly for our species, Asimov’s laws remained state-of-the-art for over half a century: this despite obvious problems with the approach, some of which are explored in Asimov’s own writings (Asimov probably having formulated the laws in the first place precisely so that they would fail in interesting ways, providing fertile plot complications for his stories).23 Bertrand Russell, who spent many years working on the foundations of mathematics, once remarked that “everything is vague to a degree you do not realize till you have tried to make it precise.”24 Russell’s dictum applies in spades to the direct specification approach. Consider, for example, how one might explicate Asimov’s first law. Does it mean that the robot should minimize the probability of any human being coming to harm? In that case the other laws become otiose since it is always possible for the AI to take some action that would have at least some microscopic effect on the probability of a human being coming to harm. How is the robot to balance a large risk of a few humans coming to harm versus a small risk of many humans being harmed? How do we define “harm” anyway? How should the harm of physical pain be weighed against the harm of architectural ugliness or social injustice? Is a sadist harmed if he is prevented from tormenting his victim? How do we define “human being”? Why is no consideration given to other morally considerable beings, such as sentient nonhuman animals and digital minds? The more one ponders, the more the questions proliferate. Perhaps "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

109 " Even environments in which organisms with superior information processing skills reap various rewards may not select for intelligence, because improvements to intelligence can (and often do) impose significant costs, such as higher energy consumption or slower maturation times, and those costs may outweigh whatever benefits are gained from smarter behavior. Excessively deadly environments also reduce the value of intelligence: the shorter one’s expected lifespan, the less time there will be for increased learning ability to pay off. Reduced selective pressure for intelligence slows the spread of intelligence-enhancing innovations, and thus the opportunity for selection to favor subsequent innovations that depend on them. Furthermore, evolution may wind up stuck in local optima that humans would notice and bypass by altering tradeoffs between exploitation and exploration or by providing a smooth progression of increasingly difficult intelligence tests. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

110 " It now seems clear that a capacity to learn would be an integral feature of the core design of a system intended to attain general intelligence, not something to be tacked on later as an extension or an afterthought. The same holds for the ability to deal effectively with uncertainty and probabilistic information. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

111 " We should recognize that there can exist instrumentally powerful information processing systems - intelligent systems - that are neither inherently good nor reliably wise. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

112 " Once villainy has had an unguarded moment to sow its mines of deception, trust can never set foot there again. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

113 " The existence of several examples of intelligence designed under these constraints should give us great confidence that we can achieve the same in short order. The situation is analogous to the history of heavier than air flight, where birds, bats and insects clearly demonstrated the possibility before our culture mastered it. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

114 " do you get when you cross an optic with a mental object? An eye-dea "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

115 " Lie detectors (should it prove feasible to make ones that are reliable and easy to use) could reduce the scope for deception in human affairs.79 Self-deception detectors might be even more powerful.80 "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

116 " Even without newfangled brain technologies, some forms of deception might become harder to practice thanks to increased availability of many kinds of data, including reputations and track records, or the promulgation of strong epistemic norms and rationality culture. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

117 " The creationist, too, is a truthlover, albeit one whose affections are unreciprocated. "

― Nick Bostrom ,

118 " an artificial intelligence need not much resemble a human mind. AIs could be—indeed, it is likely that most will be—extremely alien. We should expect that they will have very different cognitive architectures than biological intelligences, and in their early stages of development they will have very different profiles of cognitive strengths and weaknesses "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

119 " One solitary genius working out of a cork-lined bedroom can write In Search of Lost Time. Could an equivalent masterpiece be produced by recruiting an office building full of literary hacks? "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies

120 " Even within the range of present human variation we see that some functions benefit greatly from the labor of one brilliant mastermind as opposed to the joint efforts of myriad mediocrities. "

― Nick Bostrom , Superintelligence: Paths, Dangers, Strategies