# Superintelligence, Bostom > Source: https://ollybritton.com/notes/books/superintelligence/ · Updated: 2024-10-04 · Tags: notes, superintelligence, book, safe-to-post-online ![superintelligence.jpg](https://ollybritton.com/assets/attachments/img/superintelligence.jpg) > Either build AGI and destroy all of humanity or build AGI and colonise $4\times 10^{20}$ stars. This book was really, really good. I've been interested in the idea of AI safety for a while now after reading [LessWrong](https://lesswrong.com) and this book was just full of so much good information and new perspectives. ## Flashcards ### Past developments and present capabilities #### Growth models and big history * A few hundred thousand years ago, it took on the order of a million years to increase civilisations ability to support an extra one million citizens living at subsistence level. * Then it took 2 centuries. * Now it takes 90 minutes. * Another revolution on the scale of the industrial revolution could mean the world economy doubling every 2 weeks. #### Great expectations * Sometimes a problem that initially looks hopelessly complicated (creating superintelligence) turns out to have a surprisingly simple solutions (but most of the time, this isn't the case). #### Seasons of hope and despair *No highlights*. #### State of the art * Donald Knuth: "AI has by now succeeded in doing essentially everything that requires 'thinking' but has failed to do most of what people and animals do 'without thinking'". * An AI-Complete problem: a problem whose difficulty is roughly equivalent to creating a generally intelligent AI. * Copernican Revolution -- astronomy seemed hopelessly complicated when we thought we were at the center of the earth, until a paradigm shift occurred. Could be be tackling AI in the wrong way? * 2010 Flash Crash: a trillion dollars were wiped off the market due to the complex and unexpected interactions of individually simple trading components. #### Opinions about the future of machine intelligence *No highlights*. ### Paths to superintelligence #### Artificial intelligence * We know that blind evolutionary processes can produce human-level general intelligence, since it's happened at least once. * But we don't have to rely on it being "blind", a specially crafted approach could get us there a lot quicker. * This would be computationally unfeasible to carry out, but it does provide us with an upper bound on the difficulty. #### Whole brain emulation * We learnt to fly after ditching nature and running careful experiments in wind tubes. * We learnt combustion after observing naturally occurring fires. * What will the path to superintelligence be? Artificial or natural? * The WBE approach looks to create superintelligence by simulating human brains. #### Biological cognition * This approach is about augmenting our existing biological cognition, like if there was a magical smart drug that everyone could take and it would dramatically boost their IQ. * This is not revolutionary, the author drank coffee and chewed nicotine gum to help write the book. * There are already low-hanging fruit with biological cognition, such as fixing iodine deficiency in pregnant woman and infants: [Iodine and Adult IQ Meta-analysis](deficincy ). * It's not likely that there will be a magical smart drug however due to how delicately balanced and complex the human brain is. It's much more likely that a careful approach will need to be taken rather than flooding the brain with a certain type of chemical. * Another approach is embryo selection. * Embryo selection is already used in practice to select against genetic diseases that the parents of the embryos have, such as Huntington's disease. * In this new approach, embryos would be selected according to genetic markers of intelligence. * 10 generations of selecting 1 out of 10 embryos could lead to an IQ boost of 130 points. * Techniques could be used to get sperm and egg cells directly from the embryos instead which would prevent the need for the selected embryos to grow up. * This is ignoring the ethical objections, though perception could change. 18% of Americans said they wouldn't use IVF and then 53% they would once the first IVF baby was born. * Other ideas: vats full of cortical tissue, large animal brains enhanced with human genes. #### Brain-computer interfaces * Brain-computer interfaces would be complex to set up, and to get them to a level that had seamless integration of information in conciousness would likely be an AI-complete problem. * It's not about how fast information #### Networks and organizations #### Summary ### Forms of superintelligence #### Speed superintelligence #### Collective superintelligence #### Quality superintelligence #### Direct and indirect reach #### Sources of advantage for digital intelligence ### The kinetics of an intelligence explosion #### Timing and speed of the takeoff #### Recalcitrance #### Optimization power and explosivity ### Decisive strategic advantage #### Will the frontrunner get a decisive strategic advantage? #### How large will the successful project be #### From decisive strategic advantage to singleton ### Cognitive superpowers #### Functionalities and superpowers #### An AI takeover scenario ### The superintelligent will #### The relation between intelligence and motivation #### Instrumental convergence ### Is the default outcome doom? #### Existential catastrophe as the default outcome of an intelligence explosion? #### The treacherous turn #### Malignant failure modes ### The control problem #### Two agency problems #### Capability control methods #### Motivation selection methods #### Synopsis ### Oracles, genies, sovereigns, tools #### Oracles #### Genies and sovereigns #### Tool-AIs #### Comparison ### Multipolar scenarios #### Of horses and men #### Life in an algorithmic economy #### Post-transition formation of a singleton? ### Acquiring values #### The value loading problem #### Evolutionary selection #### Reinforcement learning #### Associative value accretion #### Motivational scaffolding #### Value learning #### Emulation modulation #### Institution design #### Synopsis ### Choosing the criteria for choosing #### The need for indirect normativity #### Coherent extrapolated volition #### Morality models #### Do What I Mean #### Component list #### Getting close enough ### The strategic picture #### Science and technology strategy #### Pathways and enablers #### Collaboration ### Crunch time #### Philosophy with a deadline #### What is to be done? #### Will the best in human nature please stand up ### Afterword ## Further Reading? --- Olly Britton — https://ollybritton.com. Machine-readable index: https://ollybritton.com/llms.txt