The overall risk seems to be minimized by implementing superintelligence, with great care, as soon as possible.

The stages in this sequence correspond to whole brain emulations of successively more neurologically sophisticated model organisms—for example, C. elegans → honeybee → mouse → rhesus monkey → human. Because the gaps between these rungs—at least after the first step—are mostly quantitative in nature and due mainly (though not entirely) to the differ

... See more

A superintelligent system could have disastrous effects even if it had a neutral goal and lacked self-awareness. “We cannot blithely assume,” Bostrom wrote, “that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern f

... See more